I wrote a lambda function to copy files in an s3 bucket into another s3 bucket and I need to move a very large number of these files. To try and meet the volume requirements I was looking for a way to send these requests in large batches to S3 to cut down on overhead. However I cannot find any information on how to do this in Python. There's a Batch class in the boto3 documentation but I can't make sense of how it works or even what it actually does.
1 Answers
There is no underlying Amazon S3 API call that can copy multiple files in one request.
The best option is to issue requests in parallel so that they will execute faster.
The boto3 Transfer Manager might be able to assist with this effort.
Side-note: There is no such thing as 'move' command for S3. Instead, you will need to copy, then delete. Just mentioning it for other readers.
Related