We're using Cloud Composer (aka. Airflow) for a time now. Recently we updated to the newest version composer-1.16.5-airflow-1.10.15. Ever since we're experiencing some issues surrounding so called zombie tasks.
Zombie tasks seem to arise if Airflow thinks that a task is dead, therefore killing it. The decission why a task is considered dead surrounds the heartbeat of a task, some zombie_threshold and of course worker rescources.
Usually after around 250 seconds after a task starts it is killed of by Airflow, marking it as a zombie task. But it doesn't happen on all tasks, just on some of them.
Our task usually are triggering some Dataproc/Dataflow job, so there is really no high workload on the worker. In fact GCP is telling me I should downgrade the workers to save costs, since they are not doing much. So it seemms this is no issue.
What I tried then is to set these configurations:
core-store_dag_code = "False"
core-dagbag_import_timeout = "180"
core-dag_file_processor_timeout = "180"
scheduler-job_heartbeat_sec = "5"
scheduler-scheduler_heartbeat_sec = "5"
scheduler-scheduler_zombie_task_threshold = "600"
None of these settings seem to have any effect.
Does anyone know of this issue or have some idea what could help?