Python Script as a Cron on AWS S3 buckets

user_9128435 0
Viewed 551

I have a python script which copy files from one S3 bucket to another S3 bucket. This script needs to be run every Sunday at some specific time. I was reading some of articles and answers, So I tried to use AWS lambda + Cloudwatch events. This files run for minimum 30 minutes. would it be still good with Lambda as Lambda can run max 15 minutes only. Or is there any other way? I can create an EC2 box and run it as a Cron but that would be expensive. Or any other standard way?

aws-lambda amazon-web-services amazon-ec2 amazon-s3
4 Answers

If the two buckets are supposed to stay in sync, i.e. all files from bucket #1 should eventually be synced to bucket #2, then there are various replication options in S3.

Otherwise look at S3 Batch Operations. You can derive the list of files that you need to copy from S3 Inventory which will give you additional context on the files, such as date/time uploaded, size, storage class etc.

user_2714150

The more appropriate way would be to use aws glue python shell job as it is under the serverless umbrella and you'll be charged as you go.

So this way you will only be charged for the time your code runs. Also you don't need to manage the EC2 for this. This is like an extended lambda.

user_53527480

Unfortunately, the lambda 15min execution time is a hard stop so it's not suitable for this use case as a big bang.

You could use multiple lambda calls to go through the objects one at a time and move them. However, you would need a DynamoDB table (or something similar) to keep track of what has been moved and what has not.

Another couple of options would be:

  • S3 Replication which will keep one bucket in sync with the other.
  • An S3 Batch operation

Or if its data files? you can always use AWS glue.

user_138307320

You can certainly use Amazon EC2 for a long-running batch job.

A t3.micro Linux instance costs $0.0104 per hour, and a t3.nano is half that price, charged per-second.

Just add a command at the end of the User Data script that will shut down the instance:

sudo shutdown now -h

If you launch the instance with Shutdown Behavior = Terminate, then the instance will self-terminate.

user_1747770
Related