Set a default value for an execution field of airflow sensors

user_15926888 0
Viewed 92

My problem is fairly simple if you know Airflow. I would like to set a default value other then None for the execution field execution_timeout. I was wondering if that was possible and if it is how to do it. If it's not possible is the only way to do this to set execution_timeout=timedelta(hours=2) for every sensor?

Thank you.

airflow
1 Answers

You can set it once in default_args and then it will be the default values for all sensor/operators in your DAG, you can also override in a specific operator/sensor:

default_args = {
    'execution_timeout': timedelta(hours=2),
}

with DAG(
        dag_id="my_dag",
        default_args=default_args,
        ...,
) as dag:
    op1 = MyOperator(task_id="task_op_1")
    se1 = MySensor(task_id="task_sensor_1")
    op1 >> se1
    op2 = MyOperator(task_id="task_op_2")
    se2 = MySensor(task_id="task_sensor_2", execution_timeout=timedelta(hours=1)) # override
    op2 >> se2

In the above example task_op_1, task_sensor_1, task_op_2 will have execution_timeout of 2 hours while task_sensor_2 will have execution_timeout of 1 hour.

Note that for Airflow>=2.3.0 it is also possible to define global execution_timeout by setting default_task_execution_timeout in airflow.cfg with

user_146244090
Related