Clean way to sync public s3 bucket to my private s3 bucket

user_11263171 0
Viewed 81

Binance made its data public through an s3 endpoint. The website is 'https://data.binance.vision/?prefix=data/'. Their bucket URL is 'https://s3-ap-northeast-1.amazonaws.com/data.binance.vision'. I want to download all the files in their bucket to my own s3 bucket. I can:

  1. crawl this website and download the CSV files.
  2. make a URL builder that builds all the URLs and downloads the CSV files using those URLs.
  3. Since their data is stored on s3. I wonder if there is a cleaner way to sync their bucket to my bucket. Is the third way really doable?
amazon-web-services amazon-s3
1 Answers

If you want to copy it to your own s3 bucket, you can do:

aws s3 sync s3://data.binance.vision s3://your-bucket-name --no-sign-request

If you want to copy it to your own computer into your current folder (.) you can do:

aws s3 sync s3://data.binance.vision . --no-sign-request
user_2488230
Related