How to update millions of dynamodb records in an efficient way

user_7978784 0
Viewed 575

I'm trying to update a dynamodb table having some million of rows. Tried updating it with a lambda function but it eventually times out after 15mins. I want to update a single attribute value for all the records present in the table with an efficient way. Can someone suggest how to achieve this?

dynamodb-queries amazon-dynamodb python-3.x
1 Answers

This is just my 5 cents, and it's probably not a official, perfect, or even great way of doing things. But here we go:

  • Create a SQS queue
  • Have a lambda listen to messages on that queue
  • Run the lambda, have it read the first chunk of items from the database.
  • From the query you get a LastEvaluatedKey, store this in a separate dynamodb item/table
  • Put the LastEvaluatedKey to the queue
  • When the Lambda is invoked again, use the LastEvaluatedKey passed in from the queue and pass it to the ExclusiveStartKey in your query, so that your query returns the next items from DynamoDB that you haven't processed yet

This setup means your lambda will only process small chunks at a time. It also means that when things go wrong, you have your item in DynamoDB with the LastEvaluatedKey stored, which allows you to continue from where you left off.

Hopefully this was useful, good luck :)

user_260510
Related