Best way to insert data into a Bigquery table

user_11896943 0
Viewed 6585

I would like to know if there are any preferred way to insert data into a bigquery table?

I am inserting new rows to a BQ table at the end of a Cloud Composer DAG. I've used a custom operator based on the BigQueryOperator as the following:

INSERT INTO `<project>.<dataset>.<table> (col1,col2,col3)
VALUES
(ccc,ccc,ddd),
(...,...,...)

The values added to the table can be up to 500 rows (...,...,...) but it is generally around 10-100 rows.

Is there a recommended/better way to do this? I don't know if it worth passing by GCS since the "file" containing all the rows won't be big.

google-cloud-composer airflow google-bigquery
1 Answers

Note that INSERT has a cost (size of data inserted), vs a load job via GCS which will be free.

Traditionally INSERT wasn't an option, but it's now here for convenience.

If the cost is negligible, and this is the most convenient option - go for it.

user_1324380
Related