How to reduce the temporary table expiration time in BigQuery

user_10187977 0
Viewed 822

We run Spark jobs which access BigQuery. During read phase, data is being pulled from a temporary table with naming convention _sbc_*. By default, the table exipration is 24 hours. But for our usecase, retention period of 1 hour is more than enough. I was wondering is there any we can bring down the temporary table expiration duration from 24 hours to 1 hour.

Below is how we instantiate spark config,

val sparkConf = new SparkConf
   sparkConf.setAppName("test-app")
   sparkConf.setMaster("local[*]")
   sparkConf.set("viewsEnabled", "true")
   sparkConf.set("parentProject", "<parentProject>")
   sparkConf.set("materializationProject", "<materializationProject>")
   sparkConf.set("materializationDataset", "<materializationDataset>")
   sparkConf.set("credentials", "<credentials>")

Note: Temporary table is getting created in project passed for materializationProject parameter.

Spark version : spark-2.3.1

apache-spark google-bigquery
1 Answers

spark-bigquery-connector doesn't provide any option to set expiration time over the materialized views it creates during reading.

However, if you're using a specific materializationDataset for these jobs, you can directly define the default expiration time for that dataset in BigQuery. It will be applied to all tables and views created under the dataset.

bq update --default_table_expiration 3600 materializationProject:materializationDataset
user_13865510
Related