Fastest way to move objects within an S3 bucket using boto3

user_11529280 0
Viewed 691

I need to copy all files from one prefix in S3 to another prefix within the same bucket. My solution is something like:

file_list = [List of files in first prefix]
for file in file_list:
            copy_source = {'Bucket': my_bucket, 'Key': file}
            s3_client.copy(copy_source, my_bucket, new_prefix)

However I am only moving 200 tiny files (1 kb each) and this procedure takes up to 30 seconds. It must be possible to do it fasteer?

boto3 amazon-web-services amazon-s3 python
3 Answers

I would do it in parallel. For example:

from multiprocessing import Pool

file_list = [List of files in first prefix]
    
print(objects_to_download)

def s3_coppier(s3_file):
     copy_source = {'Bucket': my_bucket, 'Key': s3_file}
     s3_client.copy(copy_source, my_bucket, new_prefix)

# copy 5 objects at the same time
with Pool(5) as p:
    p.map(s3_coppier, file_list)
user_2488230

So you have a function you need to call on a bunch of things, all of which are independent of each other. You could try multiprocessing.

from multiprocessing import Process

def copy_file(file_name, my_bucket):
    copy_source = {'Bucket': my_bucket, 'Key': file_name}
    s3_client.copy(copy_source, my_bucket, new_prefix)

def main():
    file_list = [...]

    for file_name in file_list:
        p = Process(target=copy_file, args=[file_name, my_bucket])
        p.start()

Then they all can start at (approximately) the same time, instead of having to wait for the last file to complete.

user_34626710

So I did a small experiment on moving 500 small 1kB files from the same S3 bucket to the same Bucket 3, running from a Lambda (1024 MB ram) in AWS. I did three attempts on each method.

Attempt 1 - Using s3_client.copy: 31 - 32 seconds

Attempt 2 - Using s3_client.copy_opbject: 22 - 23 seconds

Attempt 3 - Using multiprocessing, Pool (the answer above): 19 - 20 seconds

Is it possible to do it even faster?

user_115292800
Related