My Google Cloud Dataflow job is failing about an hour in with this message:
2022-03-08 12:59:23.337 PSTWorkflow failed. Causes: S08:group_by_airport/Read+group_by_airport/GroupByWindow+events_and_stats+events_to_features+into_global+shuffle/AddRandomKeys+shuffle/ReshufflePerKey/Map(reify_timestamps)+shuffle/ReshufflePerKey/GroupByKey/Reify+shuffle/ReshufflePerKey/GroupByKey/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: ch11traindata-03081130-cvo9-harness-xnps Root cause: The worker lost contact with the service., ch11traindata-03081130-cvo9-harness-qp67 Root cause: The worker lost contact with the service., ch11traindata-03081130-cvo9-harness-lrq9 Root cause: The worker lost contact with the service., ch11traindata-03081130-cvo9-harness-5mwf Root cause: The worker lost contact with the service.
Why would the worker lose contact with the service?