How to speed up download of millions of files from AWS S3

user_11948823 0
Viewed 1667

I've been trying to download these files all summer from the IRS AWS bucket, but it is so excruciatingly slow. Despite having a decent internet connection, the files start downloading at about 60 kbps and get progressively slower over time. That being said, there are literally millions of files, but each file is very small approx 10-50 kbs.

The code I use to download the bucket is:

aws s3 sync s3://irs-form-990/ ./ --exclude "*" --include "2018*" --include "2019*

Is there a better way to do this?

Here is also a link to the bucket itself.

amazon-web-services amazon-s3
1 Answers

My first attempt would be to provision an instance in us-east-1 with io type EBS volume of required size. From what I see there is about 14GB of data from 2018 and 15 GB from 2019. Thus an instance with 40-50 GB should be enough. Or as pointed out in the comments, you can have two instances, one for 2018 files, and the second for 2019 files. This way you can download the two sets in parallel.

Then you attach an IAM role to the instance which allows S3 access. With this, you execute your AWS S3 sync command on the instance. The traffic between S3 and your instance should be much faster then to your local workstation.

Once you have all the files, you zip them and then download the zip file. Zip should help a lot as the IRS files are txt-based XMLs. Alternatively, maybe you could just process the files on the instance itself, without the need to download them to your local workstation.

General recommendation on speeding up transfer between S3 and instances are listed in the AWS blog:

user_2488230
Related