I am trying to update an existing pipeline based on this manual(https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline). The first update goes through without any issues, the second one, however, always fails. There is no difference in these two updates - I am just changing a parameter in the deploy command (the pii_fields parameter).
The pipeline code is here https://pastebin.com/cRTjhNkg, the command I use for deploying looks like this:
python dev_radim_dataflow_gcs_direct_clean.py ^
--project=<PROJECT_ID> ^
--region="europe-west3" ^
--input_subscription=projects/<PROJECT_ID>/subscriptions/dev-radim-dataflow-local ^
--output_path_gcs=gs://dev_radim/dataflow_dest_local/GCS ^
--output_table_bq=<PROJECT_ID>:dev_radim.dataflow_test_b ^
--output_table_bq_schema=species:STRING,name:STRING,age:STRING,colour:STRING,publish_time:TIMESTAMP ^
--rule_key=species ^
--rule_value="guinea pig" ^
--runner=DataflowRunner ^
--window_size=1 ^
--pii_fields=name
--temp_location=gs://dev_radim/dataflow_temp_local/
The update command looks like this:
python dev_radim_dataflow_gcs_direct_clean.py ^
--project=<PROJECT_ID> ^
--update ^
--job_name="<JOB NAME>" ^
--region="europe-west3" ^
--input_subscription=projects/<PROJECT_ID>/subscriptions/dev-radim-dataflow-local ^
--output_path_gcs=gs://dev_radim/dataflow_dest_local/GCS ^
--output_table_bq=<PROJECT_ID>:dev_radim.dataflow_test_b ^
--output_table_bq_schema=species:STRING,name:STRING,age:STRING,colour:STRING,publish_time:TIMESTAMP ^
--rule_key=species ^
--rule_value="guinea pig" ^
--runner=DataflowRunner ^
--window_size=1 ^
--pii_fields=name,age,colour ^
--temp_location=gs://dev_radim/dataflow_temp/
When I run it for the second time, I am getting back INFO:apache_beam.runners.dataflow.dataflow_runner:2021-03-04T12:42:04.147Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: The new job is not compatible with <JOB_ID>. The original job has not been aborted.
Do you have any idea why would the update be successful for the first time, but fail for the second time? And additionally, is there a way how to check what specific part is not compatible?
EDIT: I believe there is some optimization happening after the first update (fusing steps) which causes the second update to fail.