Google BigQuery/Cloud Storage Invalid source uri prefix error

user_16463140 0
Viewed 947

We are currently observing a bug (at least we hope it's nothing on our end) that prevents BigQuery from accessing data stored in Google Cloud Storage (external table) using Hive Partitioning URI Prefix.

Our setup was working for months without any change, and since yesterday we are just getting an error

Error code 3 : Invalid value: Invalid source uri prefix:...

when our scheduled query tries to run.

We can access the data without partitioning but are running in an error with Hive Partitioning URI Prefix.

Is anyone else seeing such behavior?

Cheers

google-cloud-storage google-bigquery
2 Answers

AFAIK This problem is consistently occurring on and only on partitioned EXTERNAL TABLEs that have the following field/value definition (bigquery.v2.table) objectsPrefix: /*.

For our organisation this definition occurs when there is no objects prefix and the partitionsPrefix is available at the root of the bucket. For historical reasons, this impacts about 10 of ~100 EXTERNAL TABLE definitions. And as a consequence, is fairly easy to remediate manually.

Our temporary solution it to move the source data into a subdirectory (prefix) in the bucket and updating the table definitions accordingly.

This has worked so far. But we are obviously raising a support ticket with Google AND hoping to identify a real fix or the real problem as this doesn't feel like the right course of action.

All the same, I wanted to add it to this chat in case it helps anyone else temporarily. And to track this chat in case a longer term solution is found.

user_136735570

The issue with Google BigQuery has been resolved for all affected projects as of Tuesday, 2021-07-20 08:20 US/Pacific.

user_163159810
Related