S3 bucket to bucket copy performance

Lakshya Srivastava 0
Viewed 41

I'm trying to copy some files from one bucket to another (same region), getting speed of around 315mb/s. However I'm using it in lambda and there is a 15 min timeout limit. So for bigger files goes into timeout

Below is the code snippet I'm using (in python), is there any other way I can speed it up? any inputs are welcome.

 s3_client = boto3.client(
    's3',
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_key,
    aws_session_token=session_token,
    config=Config(signature_version='s3v4')
)

s3_client.copy(bucket_pair["input"], bucket_pair["output"]["Bucket"],
                               bucket_pair["output"]["Key"])

I saw many posts of passing chunksize and all, but I don't see them in ALLOWED_COPY_ARGS. Thanks.

amazon-s3 boto3 aws-lambda python-3.x
1 Answers

You can use a step function and iterate over all objects and copy them. To increase throughput you can use a map task

https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-map-state.html

If you don’t want to use stepfunction you can use one producer lambda to write all objects into a sqs queue and consume them from a lambda to copy them to the respective target.

A different option would be to use S3 object replication

https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html

But I’m not sure if that fits for your use case

emilio0
Related