BigQuery: Jobs resource are 200 OK and "outputRows": 584, but Data is not in BigQuery Table

user_7895505 0
Viewed 66

I've a Python script which Download files from a number of resources, then these files are uploaded in Google Cloud Storage Bucket after some processing, From there these files are loaded in Relevent BigQuery Tables.

i have a cronjob running this script every 15 minutes, works perfectly, but some times, like once in a week , i face a problem with only one file out of 16.

Issue is that everything seems perfect from script's log. file get downloaded processed uploaded to a bucket loaded into Bigquery. When i check job resources from Google rest v2 jobs get (https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get) status is 200 OK and "outputRows": 584, but Data is not in BigQuery Table

when i re upload that file in my resource from where the file is downloaded then file get uploaded and data can be seen in BigQuery Table.

Can any one Help me out with this?

google-bigquery google-cloud-platform python
1 Answers

(BigQuery Dev here - not enough points to comment) My hypothesis is that maybe your jobs are overwriting each others output. We will be able to get to the bottom of this if you file a bug at Issue Tracker immediately after you hit this issue with the project id and job id of the job that you found to be successful but could not find the data that it was supposed to load. Please also include details about what data you expected there and how you verified that it was not there (e.g. if you ran a query, include the job id of the query as well).

user_44965320
Related