My streaming dataflow job (from Pub/Sub source) throws multiple error messages from the worker:
Failed to read inputs in the data plane. Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 581, in _read_inputs for elements in elements_iterator: File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 706, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "keepalive watchdog timeout" debug_error_string = "{"created":"@1619783831.069194941","description":"Error received from peer ipv6:[::1]:12371","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"keepalive watchdog timeout","grpc_status":14}" >
which leads to another error message:
Python sdk harness failed: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 155, in main SdkHarness( File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 256, in run for work_request in self._control_stub.Control(get_responses()): File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 689, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "keepalive watchdog timeout" debug_error_string = "{"created":"@1619786284.366500317","description":"Error received from peer ipv6:[::1]:12371","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"keepalive watchdog timeout","grpc_status":14}" >
judging from "description":"Error received from peer ipv6:[::1]:12371","file":"src/core/lib/surface/call.cc" line, it seems that for some reason, the job cannot reach the grpc backend of the dataflow service.
Is that correct or any other reason for that error message? I use Apache Beam SDK 2.28 for python 3.8. I also use VPC settings that set using --subnetwork=https://www.googleapis.com/compute/v1/projects/.../regions/.../subnetworks/... pipeline option if that worth mentioning