Google bigquery export table to multiple files in Google Cloud storage and sometimes one single file

user_9623730 0
Viewed 2214

I am using Bigquery python libraries to export data from Bigquery tables into GCS in csv format.

I have given a wildcard pattern assuming some tables can be more than 1 GB

Sometimes even though table is few MB it creates multiple files and sometimes just it creates just 1 file.

Is there a logic behind this?

My export workflow is the following:

project = bq_project dataset_id = bq_dataset_id table_id = bq_table_id     
bucket_name =bq_bucket_name workflow_name=workflow_nm 
csv_file_nm=workflow_nm+"/"+csv_file_prefix_in_gcs+'*'client = 
bigquery.Client() destination_uri = "gs://{}/{}".format(bucket_name, 
csv_file_nm) dataset_ref = client.dataset(dataset_id, project=project) 
table_ref = dataset_ref.table(table_id) destination_table = 
client.get_table(dataset_ref.table(table_id)) configuration = 
bigquery.job.ExtractJobConfig() configuration.destination_format='CSV' – 
csv_file_nm=workflow_nm+"/"+csv_file_prefix_in_gcs
google-cloud-storage google-bigquery google-cloud-platform
2 Answers

I think this is an intended behaviour of the export. The Bigquery Export documentation specifies the following:

When you export data to multiple files, the size of the files will vary.

This corresponds to the behavior you are seeing in your exports.

user_115049090

I faced the same empty files issue when using EXPORT DATA.

After doing a bit of R&D found the solution. Put LIMIT xxx in your SELECT SQL and it will do the trick.

You can find the count, and put that as LIMIT value.

SELECT ....

FROM ...

WHERE ...

LIMIT xxx

user_174490880
Related