AWS DynamoDB streams: Can I keep records for longer than 24 hours?

user_5681624 0
Viewed 1522

I have a dynamoDB stream and lambda trigger for a table. The lambda trigger basically sync dynamoDB table to DocumentDB.

What if, DocumentDB is down for more than 24 hours. How can I put back the all the activity(put, delete, update) happened in dynamoDB back to the stream so that lambda trigger can access the records and sync the data to DocumentDB.

I see that dynamoDB stream keeps the record for maximum of 24 hours.

disaster-recovery amazon-dynamodb-streams amazon-dynamodb amazon-web-services
1 Answers

By default its not possible unlike Regular Kinesis, which has max retention of 7 days, Kinesis behind DynamoDB has max retention of 24 hours and messages will be discarded after it exceeds max retry attempts and deleted after 24 hours.

So, we need to build an exception handling process, one such methods

  • Create an SQS Queue with higher MessageRetentionPeriod (max 14 days) and set a RedrivePolicy maxReceiveCount on no of times to retry.
  • Setup Destination on Failure on Lambda to SQS.
  • Same Lambda can be slightly modified to read either from Kinesis or from SQS or a different Lambda can be used to read from SQS.

Throw error back from Lambda when it fails to write to DocumentDb. This will send record back to Kinesis/SQS. This way we can get away up to 14 days. We can add a DLQ on SQS to another SQS too, which can send left over messages after 14 days to DLQ, with destination to a persistent storage.

user_101868080
Related