how to limit the size of the file that exporting from bigquery to gcs?

user_3065606 0
Viewed 2636

I Used the python code for exporting data from bigquery to gcs,and then using gsutil to export to s3!But after exporting to gcs ,I noticed the some files are more tha 5 GB,which gsutil cannnot deal?So I want to know the way for limiting the size

google-cloud-storage google-bigquery
2 Answers

So after the issue tracker, the correct way to take this is.

  1. Single URI ['gs://[YOUR_BUCKET]/file-name.json']

Use a single URI if you want BigQuery to export your data to a single file. The maximum exported data with this method is 1 GB.

Please note that data size is up to a maximum of 1GB, and the 1GB is not for the file size that is exported.

  1. Single wildcard URI ['gs://[YOUR_BUCKET]/file-name-*.json']

Use a single wildcard URI if you think your exported data set will be larger than 1 GB. BigQuery shards your data into multiple files based on the provided pattern. Exported files size may vary, and files won't be equally in size.

So again you need to use this method when your data size is above 1 GB, and the resulting files size may vary, and may go beyond the 1 GB, as you mentioned 5GB and 160Mb pair would happen on this method.

  1. Multiple wildcard URIs

    ['gs://my-bucket/file-name-1-*.json', 'gs://my-bucket/file-name-2-*.json', 'gs://my-bucket/file-name-3-*.json']

Use multiple wildcard URIs if you want to partition the export output. You would use this option if you're running a parallel processing job with a service like Hadoop on Google Cloud Platform. Determine how many workers are available to process the job, and create one URI per worker. BigQuery treats each URI location as a partition, and uses parallel processing to shard your data into multiple files in each location.

the same applies here as well, exported file sizes may vary beyond 1 GB.

user_2437820
Related