Why WriteToBigQuery doesn't show any errors?

Question

Finally I've managed to upload data to BQ after discovering that I had incorrect schema. However, it was very hard to debug, because I didn't have any logs with my DirectRunner. How I can debug writes WriteToBigQuery when I have eg. wrong schema?

My code:

lines = messages | 'decode' >> beam.Map(lambda x: x.decode('utf-8'))
  output = ( lines
           | 'process' >> beam.FlatMap(lambda xml: [jsons.dump(model) for model in process_xmls(xml)])
           | beam.WindowInto(window.FixedWindows(1, 0)))

  output | 'Write to BiqQuery' >> beam.io.WriteToBigQuery(
            table='dataflow.test_V1',
            schema=fp_schema,
            create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
            write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)

user_582333 · Answer

The beam.io.WriteToBigQuery PTransform returns a dictionary whose BigQueryWriteFn.FAILED_ROWS entry contains a PCollection of all the rows that failed to be written. The errors themselves are logged at https://github.com/apache/beam/blob/release-2.13.0/sdks/python/apache_beam/io/gcp/bigquery.py#L861 so should show up in the worker logs.

Why WriteToBigQuery doesn't show any errors?

1 Answers