Pub/Sub to BigQuery template

user_456157 0
Viewed 320

Currently evaluating stream process from pub/sub to bq by using template that provided by Google and keep getting this error:

enter image description here

Schema of the message in the BQ is matched. But somehow job is not able to write message to BQ. Any ideas what could be the issue on this?

google-cloud-dataflow google-bigquery
1 Answers

It looks like an information error because you get an error while inserting the data into BigQuery.

You need to validate these things:

  • Some fields are missing.
  • The type of the data you are inserting mismatches with the table schema.
  • A mandatory field has no data.
  • A date does not have the correct format.

You can execute this code in the terminal to get more information about this error.

bq --format=prettyjson show -j <JobID>

It will return a JSON with more details. You can see this example:

"message": "Error while reading data, error message: Could not parse '16.66666666666667' as int for field Course_Percentage (position 46) starting at location 1717164"

You can see this document about troubleshooting streaming inserts.

You can use this option: go to the “Logging” section, then “Logs Explorer” in the console, and see more details about errors.

enter image description here

You can filter the error by product name, in this case “Dataflow” or “Bigquery”.

enter image description here

Use another filter by hour.

enter image description here

You’ll see all the errors by product, and if you click each one, you’ll see more details about this specific error.

enter image description here

You can see this documentation about logging.

user_169291450
Related