The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h

user_7369529 0
Viewed 3295

My dataflow batch job not end in 5 hours. still canceling.

Im running this type of job in scheduler every 10 min.

normally, it is finished in 10 min.

but it tooks over 5 hour!

My job is

2018-08-26_13_30_17-1172470278423820020

Error log is here

Stackdriver
2018-08-27 (06:33:14) Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been se... 2018-08-27 (08:34:08) Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been se... 2018-08-27 (10:34:58) Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been se... Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. You can get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.

google-cloud-dataflow
4 Answers

Generally The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h can be caused by too long setup progress. Just increase worker resources (via --machine_type parameter) to overcome the issue.

In my case I was installing several dependencies that required building wheels (pystan, fbprophet) and it took more than an hour on the minimal machine (n1-standard-1 with 1 vCPU and 3.75GB RAM). Using more powerful instance (n1-standard-4 which has 4 times more resources) solved my problem.

It is possibly not the case if there is single job stuck but may help someone else getting the same error.

user_112226620

Such situation can happen in three major cases: a) Some task takes more than an hour to process. b) Some task got stuck processing.

This is generally caused by transforms that take too long to process, or enter blocking state.

Best way to debug is to check for previous logs and see if there were any errors, or unexpected state. It seems that you have tried to re-run the job and it still failed. In this case you can add extra logs to the step that got stuck and see which data it got stuck at.

c) There is failure on Apache Beam / Dataflow side. This is a rare case. Please, create support ticket if you believe this is the issue.

user_116994010

I just recovered from this issue.

As mentioned in other responses, this error is due to long setup progress. This is probably the incapability of pip to resolve the dependencies in the setup.py file. It does not fail; it tries in vain "forever" to find suitable versions leading to a timeout.

  1. Recreate the environment locally using the necessary packages.
  2. Mark the versions that satisfy the dependencies.
  3. Define these versions explicitly in the setup.py.
user_43712390

I don't know why this worked but updating the package dependencies in setup.py got the job to run.

dependencies = [
    'apache-beam[gcp] == 2.25.0',
    'google-cloud-storage == 1.32.0',
    'mysql-connector-python == 8.0.22',
]```
user_30644620
Related