Wait for all the downstream tasks from the previous run in Apache Airflow

user_3929393 0
Viewed 1462

I am running multiple instances. So, max_active_runs=1 is not a solution for me. wait_for_downstream can wait only for the immediately downstream task.

From baseoperator.py code,

wait_for_downstream: when set to true, an instance of task X will wait for tasks immediately downstream of the previous instance of task X to finish successfully before it runs. This is useful if the different instances of a task X alter the same asset, and this asset is used by tasks downstream of task X. Note that depends_on_past is forced to True wherever wait_for_downstream is used. Also note that only tasks immediately downstream of the previous task instance are waited for; the statuses of any tasks further downstream are ignored.

How to wait for all the downstream tasks?

There are multiple parallel steps in my DAG.

enter image description here

EDIT:

I tried ExternalTaskSensor as suggested by Relic16 to wait for C1/C2/C3 .. to get completed. It's creating a race condition during backfill. As there are multiple instances running at a time though depends_on_past is True. As soon as 1st run's ExternalTaskSensor is in SUCCESS and the actual next task(A1/A2/A3.. ) is processed, it triggers ExternalTaskSensor in the 2nd dag run and it consumers all the slots (as there are more than 16 parallel steps) and keep waiting for C1/C2/C3 .. to get completed.

airflow
1 Answers

Let's assume B, C and D are downstream tasks of A. I can create a dummy task E which is downstream task for B, C and D.

Now I set an ExternalTaskSensor in A which waits for previous execution of task E to be finished.

For the very first run, I will mark the task successful manually but after that A will always wait for successful execution of task E from previous batch.

I can, using Trigger condition in E, further control dependency of status of B, C and D on task E.

user_111333890
Related