I'm working on a project for integration different data sources into Google BigQuery. It is the batch approach. We are using Apache Airflow for orchestration.
Simplified flow is: create raw tables (predefined DDL) -> call python code to do batch insert (via bq python client) -> trigger different SQL for transformations -> end
For testing purposes, we're using the GCP dev project.
But, recently, I found BigQuery Emulator: BigQuery Emulator. Python client example works just fine: BigQuery Emulator: Call endpoint from python client
I'm curious about how to configure the local instance of Airflow to use this emulator.
I didn't find a way to point BigQueryExecuteQueryOperator to use the emulator. We are using this operator to trigger all our SQL.
I tried to set 'gcp_conn_id' but it always fails with "Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started"
As connection type I tried HTTP and Google Bigquery, no difference.
Airflow version: 2.3.4
bigquery-emulator: 0.1.11