Do a task regardless of upstream task status

user_1195001 0
Viewed 468

I have a relatively simple DAG defined as follows:

set_variables >> create_conf_file >> check_running_job >> transform_data >> update_variables
create_conf_file >> remove_conf_file
check_running_stat_job >> remove_conf_file
transform_openidm_data >> remove_conf_file
update_orc_file_variable >> remove_conf_file

My goal here is to ensure that remove_conf_file is always executed regardless of the status of all previous tasks.

I've tried to use trigger_rule=TriggerRule.ALL_DONE in my PythonOperator call, but remove_conf_file is only executed if all previous tasks are done.
If task check_running_stat_job fails, the remove_conf_file task won't be executed.

I want the file to be removed whatever the status of upstream tasks is, DONE, FAILED, NOT DONE.

I've tried several DAG configurations but none seems to work.

[EDIT]
Here's the DAG tree view and DAG view in Airflow:

DAG standard view

DAG tree view

airflow directed-acyclic-graphs
1 Answers

I've replicated the task structure displayed in your image and successfully ran the task remove_conf_file after task spark_etl failed:

enter image description here

The key was adding trigger_rule='all_done' to the remove_conf_file task. This doesn't mean that all upstream tasks need to be executed successfully, just that they need to be finished (irrespective of success or failure). I've used BashOperator instead of PythonOperator you mention in your question.

This git repository contains corresponding dockerfile with DAG.


EDIT:

For the reference, here is what the successful code looks like if we want to test failing of spark_etl task and subsequent successful execution of remove_conf_file task:

t4 = BashOperator(
    task_id='spark_etl',
    bash_command='exit 123"',          # simulation of task failure
    dag=dag)
    
t6 = BashOperator(
    task_id='remove_conf_file',
    bash_command='echo "Task 6"',
    dag=dag,
    trigger_rule='all_done')

(Full code can be found in my git repository mentioned above.)

user_141437640
Related