Move 2000 objects from one S3 bucket to another

Question

I need to move around 2000 S3 objects from one S3 bucket to another in the same region and in the same AWS account. I have a list of S3 keys which I need to move (Not every object in the bucket). If I'm using a forEach loop for each key and use s3Client.copyObject() then the request times out after 30 seconds and it is able to copy around 200 objects. What should I do in this situation.

My code looks like this

List<String> s3Keys;   // size of list is 1000
s3Keys.forEach(key -> 
    s3Client.copyObject(bucket1, key, bucket2, key)
);

It keeps running for 30 sec. and then times out.

My code looks like this

List s3Keys;   // size of list is 1000
s3Keys.forEach(key -> 
    s3Client.copyObject(bucket1, key, bucket2, key)
);

It keeps running for 30 sec. and then times out.

user_982341 · Answer

You should probably try using S3 batch operations.

Move 2000 objects from one S3 bucket to another

1 Answers