List all the tables in a dataset in bigquery using bq CLI and store them to google cloud storage

user_2529269 0
Viewed 13323

I have around 108 tables in a dataset. I am trying to extract all those tables using the following bash script:

# get list of tables
tables=$(bq ls "$project:$dataset" | awk '{print $1}' | tail +3)

# extract into storage
for table in $tables
do
    bq extract --destination_format "NEWLINE_DELIMITED_JSON" --compression "GZIP" "$project:$dataset.$table" "gs://$bucket/$dataset/$table.json.gz" 
done

But it seems that bq ls only show around 50 tables at once and as a result I can not extract them to cloud storage.

Is there anyway I can access all of the 108 tables using the bq ls command?

gcloud-cli google-bigquery google-cloud-platform
2 Answers

The default number of rows when listing tables that bq ls will display is 100. You can change this with the command line option --max_results or -n.

You can also set the default values for bq in $HOME/.bigqueryrc.

Adding flags to .bigqueryrc

user_80167200

I tried with CLI and This command worked for me:-

bq ls --max_results 1000 'project_id:dataset'

Note: --max_results number_based_on_Table_count

user_75347900
Related