aws copy speed advantage of using wildcards versus sending all commands?

user_1906543 0
Viewed 185

I am trying to copy a large number of files between buckets, and am only getting around 15 files per second. That is not usable, with 500k files...

So I was wondering if it actually makes any difference to use a wildcard in a cp statement, as opposed to sending individual cp statements? What is the "standard" to use here? Or are both resulting in the same client-side and server load?

as an example, I have now written code to group files based on their batch id and send them in groups. But I do not get the impression (from a very basic test) that it is faster?

e.g.,

aws s3 cp <path>/XY.15937610001 <path_to>
aws s3 cp <path>/XY.15937610002 <path_to>
aws s3 cp <path>/XY.15937610003 <path_to>
:
aws s3 cp <path>/XY.15937615999 <path_to>

versus:

cmd

aws s3 cp <path> <path_to> --recursive --exclude="*" --include="XY.159376*"

thank you

PS edit - is the only way to speed this up, using max_concurrent_sessions, or something like S3DistCp (s3-dist-cp) (whatever that may be)? Both options are not available to me currently...

cp amazon-web-services amazon-s3
1 Answers

I believe some of the time of using the --include/--exclude flag will be spent querying for any specific object with a key that has that pattern?

Can you batch run a number of prefixes in parallel specifying the exact prefix?

Whenever the query is performed to list objects, the backend API is having to check the contents across many distributed nodes in the S3 architecture to check if they match this condition.

If you could run these in parallel then you might see some speed increase, although be aware of the following factors that can slow down:

  • Disk I/O - If writing to disk, be aware that the disk may throttle. With GP2 volumes in EBS they use burstable credits for IOPs performance.
  • Network I/O - Can the server support that throughput? If you're using an EC2 instance then look at instance types.
  • EBS Optimized - Ensure your instance has EBS optimized enabled, if you're using an EC2. This helps to avoid contention of bandwidth between network I/O and storage I/O on your EBS volume.
  • VPC Endpoints - Setup a VPC endpoint for S3 and apply it to the route table of your instance. By doing this the connection to S3 becomes private, reducing the latency both for API actions and copying.
user_134609330
Related