Says I have composer airflow run under GCP project named project-central (as source project). I have a lot of task instances using BigqueryOperator to generate a table for multiple projects (as target table), for example:
- project-1
- project-2
- project-3
From what I understand, BigqueryOperator by default will run SQL using bq slot from which composer is running (based on the example above project-central).
With the aim of:
- to split the resources,
- to prevent the hit limit of bq slot in project-central, and
- to have cost attribution
I need to have the task instance BigqueryOperator running using bq slot of the target table. So if the target table is project-2, BigqueryOperator uses bq slot from project-2 instead of project-central. Is it possible to have those settings in BigqueryOperator params?
I see params bigquery_conn_id (str) – reference to a specific BigQuery hook from this link https://airflow.apache.org/docs/apache-airflow/1.10.6/_api/airflow/contrib/operators/bigquery_operator/index.html
But is it correct to use that to set up which project to use bq slot?
If yes, what is the value of that params?
Because I can't find many examples of that, just this under the same link, and I still don't understand what that is means.
bigquery_conn_id='airflow-service-account'bigquery_conn_id='bigquery_default'