Apache Airflow pools: used slots > available slots

Viewed 4839

I have 6 subdags. Each of them contains a task with pool='crawler' that requires a lot of resources so I created a pool crawler with only 1 slot.

When I run the DAG it seems that the pool restriction is bypassed and all six tasks are executed at the same time (as you can see from the screenshot).

How can I force used slots to be <= available slots?

airflow pools

1 Answers

From the source code:

Airflow pool is not honored by SubDagOperator. Hence resources could be consumed by SubdagOperators

Related