How to pause / Un pause multiple dags in airflow

user_7741772 0
Viewed 2804

We have 100 of dags which has a prefix with "dag_EDW_HC_*" . we have below command to pause the dag

Command: airflow pause dag_id

Is there any way we can pause all the 100 dags "dag_EDW_HC_*" in a single go .. (In programmatic in python or any other way) ..?

airflow-scheduler airflow
2 Answers

The absolute easiest (and likely fastest) way I can think of is to update the database:

UPDATE  dag
   SET  is_paused = false
 WHERE  dag_id LIKE 'dag_EDW_HC%';
user_1746520

If you want to do this regularly you can create a DAG specifically for this purpose with the corresponding PythonOperator for that and specify parameters when triggering DAG. From a running task instance (in the python_callable function that we pass to a PythonOperator or in the execute method of a custom operator) you have access to the DagBag object which contains dag ids of all DAGs loaded into Airflow environment which you can use to get DagModel-s which you can loop through and pause all DAGs:

def python_callable():
   dag_bag = DagBag(read_dags_from_db=False)
   for dag_id_ in dag_bag.dag_ids:
       dag_model = airflow.models.dag.DagModel.get_dagmodel(dag_id_)
       dag_model.set_is_paused(True)

The current code is for the 2.0.1 version and it could differ for different versions of Airflow. You should check the documentation for your version of Airflow server if this callable does not work for you.

user_15186970
Related