I want to schedule aws s3 sync s3://bucket1 s3://bucket2 command to run everyday at defined time, say 3 AM.
What options do we have to schedule this using aws resources like lambda etc?
I saw many people using Windows scheduler, but as this is s3 to s3 sync, its not a better option to use Windows scheduler of servers to run this command through cli.
This sounds like a case of The X-Y Problem. That is, it's likely "scheduling an AWS CLI command to run" is not your underlying problem. I'd urge you to consider whether your problem is actually "getting one S3 bucket to exactly replicate the contents of another".
On this point, you have multiple options. These fall broadly into two categories:
Actively sync objects from bucket A to bucket B. This can be done using any number of methods already mentioned, including your idea of scheduling the AWS CLI command.
Lean on S3's built-in replication and this is probably what you want.
The reason S3 replication was implemented by AWS is to solve exactly this problem. Unless you have additional considerations (if you do, please update your question, so we can better answer it :) ) replication is likely your best, and easiest, and most reliable, option.
There are so many ways to do this, I'll elaborate on the ones I use.
Cloudwatch events to trigger whatever is going to perform your task. You can use it just like a crontab.
Lambda functions: 1 - give the lambda function an IAM role that allows read from bucket1 and write to bucket2 and then call the api. 2 - since aws cli is a python tool, you could emmbed aws cli as a python dependency and use it within your.
Here's a link to a tutorial: https://bezdelev.com/hacking/aws-cli-inside-lambda-layer-aws-s3-sync/
Docker+ECS Fargate: 0 - pick any docker image with aws-cli preinstalled like this one 1 - create an ECS Fargate cluster (will cost you nothing) 2 - create an ECS task definition and inside it use the image you chose the on step 0 and on command put "aws s3 sync bucket1 bucket2" 3 - create a schedule that will use your task definition created on step 2
Additional considerations: Those are the ones I would use. You could also have cloudwatch trigger a cloudformation that would create an ec2 instance and use the userdata field to run the sync, you could create an ami of an ec2 that on /etc/rc.local has the sync command and then a halt command, and several other options that work. But I'd advise you get the lambda option, unless your sync job takes more then 15 minutes (which is lambda's timeout) then I'd go with the docker option.
