How to know that the data in the BigQuery streaming buffer is unavailable?

user_4064443 0
Viewed 62

According to the following documentation https://cloud.google.com/blog/products/gcp/life-of-a-bigquery-streaming-insert :

In these cases you may observe that the buffered data does not contribute to the query's results. (We're working on improving visibility into instant availability contributions to query results.)

Is there a way to know from the Java API that the query that got executed was not able to read the data in the streaming buffer, so that we can reschedule this operation later on in order not to miss some data in our final consolidated table?

google-bigquery
1 Answers

That note is 5 years old and out of date. Data written to the "streaming buffer" is immediately available for querying, unless the writer of the data explicitly used a streaming mode that keeps data buffered/pending until it commits the stream.

This is aside from the fact that under the hood the system was refactored and it doesn't use the old "streaming buffer" anymore, it directly writes to the BigQuery storage API. First the data goes into write-optimized storage to support high streaming throughput, and it is later copied to read-optimized storage, to optimize query performance. But the user does not need to worry about this.

user_12639350
Related