Can I restrict max_active_runs of a DAG only for runs with the same dag_run.conf?

Viewed 680

I have a DAG that runs in a multi tenant scenario. The tenant ID get's set in the dag_run.conf when the DAG is triggered. I want to ensure that there is at most one active run per tenant at a time, but potentially many active runs simultaneously across all tenants.

So far I have found the max_active_runs setting, but that would require me to actually setup one DAG per tenant, which I am trying to avoid.

Is there a way to achieve this in airflow or am I approaching the problem in the wrong way?

1 Answers

You are using dag_run.conf which means that you are triggering your dags manually. Currently there is a bug (Airflow 2.0.1) max_active_runs isn't respected for manual runs (see GitHub issue).

Related