Airflow 2.0 Issues : Too many airflow supervisor tasks

user_6886967 0
Viewed 403

I installed airflow 2.0 using docker swarm and Celery Executor.
After 1 week, celery workers memory is overflowing by airflow task supervisor (screenshot attached)
Anyone faced such issues ? Any suggestions ?

enter image description here

airflow-2.x airflow
1 Answers

In Airflow 2.0, there are 2 ways of creating child processes.

  1. forking of the parent process (Fast)
  2. spawning a new python process using python subprocess (Slow)

By default, airflow 2.0 uses (1) method. Forking the parent process is faster. On the other hand, the child process is not killed after the task completion. Number of child processes keep increasing till the memory exhaused.

I switched to subprocess method (2) by setting execute_tasks_new_python_interpreter = True. Here, each python process is killed and everytime new process is created. This might be slow but memory is effectively utilised.

user_68869670
Related