AWS S3 migrating 50 TB of data between regions

There's nothing special about moving data across regions compared to within one region other than the addition of bandwidth fees. It's possible AWS DataSync or Amazon S3 Transfer Acceleration could impact cross-region performance, but in my testing the effect was negligible. Those technologies are more geared towards moving data between on-site storage and S3, not between two S3 regions.

Recommendation

For syncing/copying data between S3 buckets I recommend S3P

(disclaimer, I wrote it)

S3P can be up to 50x faster than aws-cli. I've sustained speeds of 8gigabytes/second within one region. I developed S3P for a client who had to move 1 petabyte of data in 10,000,000 files between S3 buckets (same region). The speed is largely determined by the average file size, in this case around 100MB. It'll also be slower across regions, but with S3P it's entirely possible you could copy 50TB in a handful of hours.

S3P gets its speed by not only doing massively parallel copying, but also massively parallel item-listing. The later is how S3P beats out other tools. If there's anything else out there that's faster, I'd love to know about it.

Easy to Try

You can try out s3p easily if you have Node.js installed, just open a terminal and run the following to get a list of commands:

npx s3p

• npm: npmjs.com/package/s3p
• source: generalui/s3p

Note: S3P is fast, even running on your local machine, but to get maximum performance, run it on a decent sized EC2 instance in the same region as one of your S3 buckets (e.g. m5.xlarge).

A few options:

• If your regions are in the same country, the Data Transfer price is quite reasonable. It will just take time.
• Rather than using the AWS CLI, consider using AWS DataSync that "makes it simple and fast to move large amounts of data online between on-premises storage and Amazon S3, Amazon Elastic File System (Amazon EFS), or Amazon FSx for Windows File Server."
• I don't think AWS Snowball will work for transferring data between regions. It is only for going in/out of S3 and your own location.

You can cross region replicate your data (CRR) To replicate existing objects, you can run the following cp command after setting up replication on the source bucket:

aws s3 cp s3://source-awsexamplebucket s3://source-awsexamplebucket --recursive --storage-class STANDARD

This command copies objects in the source bucket back into the source bucket, which triggers replication to the destination bucket.

There is also the aws s3 sync option that can be used to synchronize content between two buckets.