Error message from worker: generic::aborted: SDK harness sdk-0-1 disconnected

user_8330018 0
Viewed 1419

I am having some issue with one of Dataflow jobs. From time to time I get this error messages. It seems that after this errors, the job keeps running fine, but, this night it actually stuck, or it started to slowly process the elements. Also you can see from the screenshot that the workers started to behave very weirdly as shown in the CPU usage chart in the below screenshot.

Error message from worker: 
generic::aborted: SDK harness sdk-0-1 disconnected.   
generic::aborted: SDK harness sdk-0-1 disconnected.   
generic::aborted: SDK harness sdk-0-1 disconnected.   
generic::aborted: SDK harness sdk-0-1 disconnected.   
passed through: ==> dist_proc/dax/workflow/worker/fnapi_service.cc:631   generic::aborted: SDK harness sdk-0-1 disconnected.   
generic::aborted: SDK harness sdk-0-1 disconnected.   
passed through: ==> dist_proc/dax/workflow/worker/fnapi_service.cc:631   generic::aborted: SDK harness sdk-0-1 disconnected.   
passed through: ==> dist_proc/dax/workflow/worker/fnapi_service.cc:631 generic::aborted: SDK harness sdk-0-1 disconnected.   
passed through: ==> dist_proc/dax/workflow/worker/fnapi_service.cc:631   generic::aborted: SDK harness sdk-0-1 disconnected.   
generic::aborted: SDK harness sdk-0-1 disconnected.   
passed through: ==> dist_proc/dax/workflow/worker/fnapi_service.cc:631   generic::aborted: SDK harness sdk-0-1 disconnected.   
generic::aborted: SDK harness sdk-0-1 disconnected.

enter image description here

apache-beam google-cloud-dataflow
1 Answers

That error message could be due to a wide variety of causes, so unless the error message is accompanied by the other behavior the user described, this could be due to any number of errors.

In my case, it was due to the IO error: No space left on disk

A good way to further investigate is by viewing the

  • Diagnostics : table showing where errors occurred along the chosen timeline and possible recommendations for your pipeline.

enter image description here

user_118661330
Related