Dataflow is reprocessing input file through entire pipeline from the beginning

Pav3k 0
Viewed 46

The title of this question is just my hypothesis and I would like you to ask about it.

I have dataflow pipeline (Python) which does something like this:

  1. Put file in GCS bucket.
  2. GCS bucket sends notification to PubSub topic.
  3. PubSub topic triggers Dataflow streaming pipeline.
  4. Some DoFn transformations.
  5. Write to BigQuery.

I discovered that one file that had been processing didn't land in BQ table. No errors, all logs showed that all is fine and we came to the final step - write to BQ. After several seconds the same file started to be processed again, from the beginning.

Have you seen this mechanism in your pipelines? I have two theories:

  1. GCS send's duplicated notifications and that's why process for one file was triggered twice. This makes sense, also this is described in google documentation but this not explains why first pipeline run didn't succeed (I could not find this file in BQ target table).
  2. WriteToBiqQuery step failed (silently) and dataflow/apache beam decided to reprocess the file from the beginning. That would make sense but I can't find any prove that this mechanism is a real thing.

Any thoughts?

apache-beam google-cloud-dataflow google-bigquery python
1 Answers

I think when the file is added in the Cloud storage bucket, two events are fired.

In order to be sure to treat your file when he has finished to be added to the bucket, you need to add a filter on your Dataflow job on event type with OBJECT_FINALIZE value.

You can check the documentation for GCS to Pub Sub events and notifications :

https://cloud.google.com/storage/docs/pubsub-notifications

Mazlum Tosun0
Related