Dataflow ReadFromBigQuery transformation - How to configure location for Temporary table

user_1651397 0
Viewed 681

When we perform the ReadFromBigquery transformation, Dataflow creates a temporary dataset in which it stores the data before reading it.

My user is only allowed to create a dataset in Europe Region (not in the US), and it seems that dataflow is using the US region by default. How can I ask dataflow to create the Temp Dataset in Europe ?

Important: the tables that I am reading are in Europe and I specified "Region = Europe" in pipeline options

Please find Below:

  • The read transformation: read = ( p | 'ReadForVal' >> beam.io.ReadFromBigQuery(query='SELECT id_ligne FROM project.dataset.table', use_standard_sql = True))
  • The error: content <{ "error": { "code": 403, "message": "US violates constraint constraints/gcp.resourceLocations on the resource projects/irn-71631-lab-80/datasets/temp_dataset_243e02d1cda342d4962195b28cf33bba", "errors": [ { "message": "US violates constraint constraints/gcp.resourceLocations on the resource projects/irn-71631-lab-80/datasets/temp_dataset_243e02d1cda342d4962195b28cf33bba", "domain": "global", "reason": "policyViolation" } ], "status": "PERMISSION_DENIED" } } >

I am struggling for a few days now... Thank you a lot for your help!

apache-beam google-bigquery dataflow google-cloud-platform
1 Answers

Unfortunately there doesn't seem to be a way to manually set the location of the temporary dataset. From the code, it appears to get the location from the table in the query:

def _setup_temporary_dataset(self, bq):
  location = bq.get_query_location(
      self._get_project(), self.query.get(), self.use_legacy_sql)
  bq.create_temporary_dataset(self._get_project(), location)

And the documentation for get_query_location states "This method returns the location of the first available referenced table for user in the query".

The simplest workaround at the moment is to only read tables in Europe, if possible by copying any tables from the US over before reading them from Dataflow. Actually adding the ability to configure the temporary database would probably be welcome in Beam, so I encourage you to report this as a feature request on the Beam Jira.

user_113370330
Related