Airflow retry of multiple tasks as a group

user_15749851 0
Viewed 332

I have a group of tasks that should run as a unit, in the sense if any of the tasks from the group fail, the whole group should be marked as failed.

I want to be able to retry the group when it has failed.

For example, I have a DAG with these tasks:

taskA >> (taskB >> taskC) >> taskD

I want to say that (taskB >> taskC) is a group.

If either taskB or taskC fails, I want to be able to rerun the whole group (taskB >> taskC).

airflow-2.x airflow
1 Answers

This is a two parts question.

First, In Airflow downstream task can not effect upstream task. Assuming structure of:

taskA >> taskB >> taskC >> taskD

then if taskB is successful and taskC failed. it can not change the state of taskB to failed.

Second, clearing (rerun) a TaskGroup is a feature that currently is not available. There is an open feature request for it in Airflow repo. You can view it in this link.

user_146244090
Related