AWS lambda or AWS batch to delete rows from dynamo DB based on certain criteria

sidkool3k 0
Viewed 27

I have a requirement where I need to delete records based on criteria it doesn't need to be real time, the records are stored in dynamo DB. Can AWS lambda be scheduled ex run every day @ 11pm can I package a cron job or is it better to go with AWS batch only worry with batch is it has more overheads.

Thanks

aws-batch amazon-dynamodb aws-lambda
1 Answers

If your criteria is time then the best option is to use the Time To Live (TTL). DynamoDB TTL feature delete a records after expiry.

TTL is useful if you store items that lose relevance after a specific time. The following are example TTL use cases:

  • Remove user or sensor data after one year of inactivity in an application.
  • Archive expired items to an Amazon S3 data lake via Amazon DynamoDB Streams and AWS Lambda.
  • Retain sensitive data for a certain amount of time according to contractual or regulatory obligations.

AWS provides details in this document.

If instead you have more complex criteria, you could use Lambda as you requested, however creating a Lambda function alone won't be enough. In fact Lambda need always something to kick it off, like a cron job. AWS recommends to use CloudWatch to schedule Lambda runs. Relevant documentation can be found in the tutorial "Schedule AWS Lambda Functions Using CloudWatch Events" This process includes the following steps:

  1. Create an AWS Lambda Function to log the scheduled events. Specify this function when you create your rule.

  2. Create a rule in CloudWatch to run your Lambda function on a schedule. A rule would look like:

    aws events put-rule
    --name my-scheduled-rule
    --schedule-expression 'rate(5 minutes)'

  3. Verify the rule checking that your Lambda function was invoked.

AR10
Related