How to sync a lot of buckets from shell or python

yixuan 0
Viewed 31

I can use aws s3 ls to list buckets like this:

$aws s3 ls --endpoint http://XXXXXX
...
2022-08-22 16:47:37 cluster-back
2022-08-23 11:54:46 dog
2022-09-08 10:22:41 kesci-annotations
2022-09-08 10:22:59 kesci-datasets-ng
2022-09-08 10:23:13 kesci-datasets-ng-parsed-meta
2022-09-08 10:25:07 kesci-model-service-input
2022-09-08 10:25:39 kesci-model-service-output
2022-09-08 10:26:09 kesci-workspace-ng
2022-09-08 10:26:39 kesci-workspace-ng-parsed-meta
...

How can I use shell or python to sync all of the buckets without having to provide all of the bucket names?

I then want to use this shell or python to kubernetes`s cronjob.

synchronization amazon-web-services
1 Answers

The aws s3 sync command will copy from one source path to one destination path. The source path would include the bucket name. Therefore, you would need to run one sync command per bucket.

You could first run a command to retrieve all the bucket names:

aws s3api list-buckets --query 'Buckets[].[Name]' --output text

This will output each bucket name on a separate line.

You could then pass these bucket names into xargs to generate multiple aws s3 sync commands that will copy all the buckets.

John Rotenstein0
Related