Google Dataflow stalled after BigQuery outage

Question

I have a Google Dataflow Job running. The dataflow job is reading messages from Pub/Sub, enrich it and write the enriched data into BigQuery.

Dataflow was processing approximately 5000 messages per second. I am using 20 workers to run the dataflow job.

Yesterday it seems there was a BigQuery outage. So writing the data in BigQuery part failed. After some time, my dataflow stopped working. I see 1000 errors like below

  (7dd47a65ad656a43): Exception: java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "The project xx-xxxxxx-xxxxxx has not enabled BigQuery.",
    "reason" : "invalid"
  } ],
  "message" : "The project xx-xxxxxx-xxxxxx has not enabled BigQuery.",
  "status" : "INVALID_ARGUMENT"
}
com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.insertAll(BigQueryTableInserter.java:285)
com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.insertAll(BigQueryTableInserter.java:175)
com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.flushRows(BigQueryIO.java:2728)
com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.finishBundle(BigQueryIO.java:2685)
com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.finishBundle(DoFnRunnerBase.java:159)
com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.finishBundle(SimpleParDoFn.java:194)
com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.finishBundle(ForwardingParDoFn.java:47)
com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.finish(ParDoOperation.java:65)
com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:719)
Stack trace truncated. Please see Cloud Logging for the entire trace.

Please note that dataflow is not working even the BigQuery started working. I had to restart the dataflow job to make it work.

This causes data loss. Not only at the time of outage, but also until I notice the error and restart dataflow job. Is there a way to configure the retry option so that dataflow job does not go into stale on these cases?

Google Dataflow stalled after BigQuery outage

0 Answers