How to include DeltaLake Files from GCS to BigQuery

user_4801289 0
Viewed 1213

Is there a library/connector available to import Delta Lake files stored in Google Cloud Storage (GCS) directly to BigQuery ?

I have managed to write BigQuery tables using a Spark Dataframe as intermediary but I can't find any direct connector or BigQuery Library that does this without transitionning through spark dataframes.

Update 1: I tried using the official connector spark-bigquery-connector but documentation is lacking on how to point to a specific project in BigQuery so I couldn't go further than loading the DeltaLake files from GCS in a Dataframe.

Update 2: using Javier's comment, I managed to write to BQ but this solution isn't optimized and as much as I can optimize the spark Job, it won't be as direct as using a Google Bigquery library that does it under the hood

Update 3 and Temporary Solution: Not finding any direct solution, I ended up using spark-bigquery-connector to ingest Delta files as following :

myDeltaTable.toDF
  .write
  .format("bigquery")
  .mode("overwrite")
  .option("parentProject", "MY_PARENT_PROJECT")
  .option("project", "MY_PROJECT")
  .option("temporaryGcsBucket", "A_STAGING_BUCKET_IN_GCS") // Mandatory
  .option("partitionField", "MY_PARTITION_FIELD")
  .save("DATASET.TABLE")
delta-lake google-cloud-storage apache-spark google-bigquery google-cloud-platform
1 Answers

There is no way to ingest a Datalake file in GCS to BigQuery without going through some intermediary.

You could setup a GCE VM that downloads the Datalake file from GCS, reads it using the Datalake Standalone Connector and then write to BigQuery (either by the streaming API or writing to a supported format like Parquet and importing).

However this is essentially doing manually the same thing that Spark would be doing.

user_36453700
Related