Say I have a dag in which one task depends on 4 tasks. All the 4 tasks should only be triggered if the dag's previous run's same task was successful. So all the tasks have depends_on_past as True. However the last task of the dag is a clean up task which should always be triggered. So in the case where the previous day's task failed, the current day's task is not triggered which makes the last task also not triggered. What would be the way to address this?
1 Answers
Always set trigger_rule=TriggerRule.ALL_DONE in your cleanup tasks (those tasks which must always run, irrespective of status of upstream tasks)
And while you seem to have already figured it, it might also make sense to have depends_on_past=False set on your cleanup tasks
Related