Airflow tasks not failing if exception happens in callback function

user_18329870 0
Viewed 564

Airflow tasks not failing when an exception occurs during on_success_callback function execution even if error is caught and AirflowException is thrown in the callback func.Is this normal behaviour.

Is there any other way to make sure the task fails if an exception occurs during the callback function execution.

airflow
2 Answers

I believe when you reach the on_success callback , it means you the task has already succeeded. Now if you want the task to still fail because of the error in on_success callback then you might need to implement a try except block, where in you must set the task as failed manually. Something like this.

def on_success_callback(context):
    try:
        raise ValueError
    except:
        dag = context['dag']
        tasks = dag.task_ids
        print(context['execution_date'])
        dag.clear(
        task_ids = tasks,
        start_date = context['execution_date'],
        end_date = dag.end_date
        )

You can derive the task instance from the context and then error it out.

user_140912890

on_success_callback is executed after the task has finished with Success. Raising exceptions in on_success_callback will not result in changing the Task status.

If the code you execute in the on_success_callback suppose to fail the task in case of exception then this code should be in the task code.

user_146244090
Related