Google BigQuery bq load for certain columns?

user_50305 0
Viewed 630

I have a dataset of .orc files. I use the CLI to load my data in BQ using the bq load command in the gcloud utils. My data have 4 columns. The third column duplicates the 4th, but I'd like to keep the 4th column. Is there a way to only ingest certain colums with bq load of orc files?

google-bigquery google-cloud-platform
1 Answers

To ingest only some columns, you can specify the columns in the end of the command:

bq load --source_format=ORC DATASET.TABLE gs://YOU-FILES_LOCATION COLUMN1:TYPE, COLUMN2:TYPE, ...

Lets suppose two scenrios:

  1. In the first scenario you don't have a table in BigQuery and you want to load only some fields of you ORC file. If you run the command above specifying some fields, will be created a table with only that fields.
  2. In the second scenario, you already have a table in BigQuery with NULLABLE fields. In this case, if you run the command above specifying the fields, you will load only that fields of your data into the table and the other fields will be NULL for that rows.

Please let me know if it helps you with your problem.

user_125184870
Related