So I'm using Datastream's new feature to stream data from Cloud SQL MySQL database to BigQuery, I've set the maximum data staleness of the stream to 8 hours.
THE PROBLEM:
To initialise the stream I performed a backfill on my tables. Datastream indicated that the backfill is complete and when I go to BigQuery in the table preview and using the query SELECT * FROM mydataset.__TABLES__ the row count and size of the table correspond to the expected value.
HOWEVER
When I query the table like:
SELECT COUNT(*) FROM mydataset.table
I get a much lower row count. The correct number of rows appears later after the maximum data staleness time is exceeded (after 8 hours in my case).
MY QUESTION:
Is this behaviour normal ? Isn't the backfilled data supposed to be available in bigquery if Datastream indicated that the backfill is complete?
NOTE: this Datastream MySQL-to-BigQuery feature is still in preview
Thank you in advance.