How to handle Synchronization with the Lambda and S3 Buckets

Sandeep Kokate 0
Viewed 22

I have a lambda function written in Python, This function process some file present in the S3 bucket and once process is done, move it to the another S3 Bucket. This Lambda function is triggered with File Creation on S3 bucket and the scheduler with every 15 min. But sometimes what happens is that, same file is processed multiple times due to trigger and scheduler,

How can make it sync? If the file creation trigger is running for one file in s3 bucket, that file should be locked and scheduler can't process it. Simple term: Trigger: File Found -> Lock it -> Process it -> Unlock it

So if the scheduler get called and found same file, it can't access it due to lock.

Any help on this?

amazon-s3 aws-lambda amazon-web-services python
1 Answers

I don't have enough reputation so just comment down here. I think you are using scheduler for processing failed data is a bad design. You can have a better structure using SQS + SQS dead letter queue + Lambda. It's one of the best practices for processing events and failed events

Mink0
Related