Airflow - Load Parquet table into BigQuery

user_12160102 0
Viewed 1636

I need advice!

I'm trying to load a parquet file located in a Google Cloud Storage into BigQuery by using the Apache Airflow GoogleCloudStorageToBigQueryOperator.

My code is:

export_transactions_to_bq_task = GoogleCloudStorageToBigQueryOperator(
        task_id="export_transactions_parquet_to_bq",
        bucket="tempo-io-datalake-delivery",
        source_objects=["bigquery/atlassian/export_transactions.parquet"],
        destination_project_dataset_table="datalake-tempo-io:atlassian.transactions",
        source_format="PARQUET",
        create_disposition="CREATE_IF_NEEDED",
        write_disposition="WRITE_TRUNCATE",
        autodetect=False,
        google_cloud_storage_conn_id="google_cloud_default",
        bigquery_conn_id="bigquery_default",
        dag=export_transactions_dag
        )

The parquet format doesn't need any schema specification because it is self-describing (?!) but Airflow give me an error that one of schema_fields, schema_object or autodetect settings must be provided.

Any clues?

parquet airflow google-cloud-storage google-bigquery
3 Answers

This is a bug which has been fixed in Airflow v1.10.5 with this PR

Since the logic in the gcs_to_bq operator is having schema_object as None, schema_fields as None, and a default False value for autodetect, it will always throw that exception.

If you don't want to upgrade your Airflow to 1.10.5, setting autodetect=True would be the workaround.

user_94655610

I believe this is because your disposition is set to create_disposition="CREATE_IF_NEEDED".

You can try setting autodetect=True or change the disposition to CREATE_NEVER. If you set the autodetect to True then the schema is inferred from the parquet file and if you change the disposition to CREATE_NEVER then the schema is inferred from the Bigquery table "datalake-tempo-io:atlassian.transactions".

Let me know if that helps

Thanks, jjayadeep

user_32266300

According with your error description which is "Exception: ('BigQuery job status check failed. Final error was: %s'" the issue seems to be related with the job location and Bigquery Dataset location, as @RyanTheCoder mentioned, you should check the next question and it might solve your issue.

user_121497760
Related