I'm looking for a faster method to download a ton of objects (6.5 million in my case) from a bucket. The average object size is 2kb (it's a JSON file). The method I used was gsutil -m cp -r gs://<bucket>/<folder> . which took 14 hours for 1M objects.
It's not feasible to run this on my laptop for 7 days straight. Any ideas?
PS: I don't need them to be in individual JSON files. I'm thinking to create a script that pulls a file from the bucket, and adds a row to a CSV, then deletes the file.