How can define in Airflow dependencies in grouped tasks?

user_12111873 0
Viewed 303

i want to launch grouped tasks in airflow. When the first group end then start the second group of tasks, example:

I have task A,B,C and D and i want run tasks A and B together and when A and B will finish, then C and D will start together

Like this, but not working...

[A,B] >> [C,D]

(the tasks are BashOperator type)

could you help me??

thanks!!!

airflow control-flow directed-acyclic-graphs
1 Answers

There are two ways I will show how you can do this.

1. Define the dependencies one by one.

Now you are trying to do it all in one line. This is not possible because we are only able to set a dependency for a lists to a single task and from a single task to a list. However, it is not possible to go from a list to a list. Because your example only has 4 tasks, we can do it in two lines.

# original
# [A,B] >> [C,D]
# new way
[A, B] >> C
[A, B] >> D

2. Create a DummyOperator in the middle.

Let's introduce task E, a DummyOperator. The DummyOperator will always succeed automatically once its dependencies are all done. Now we can define it as follows.

[A, B] >> E
E >> [C, D]

In general, this is a very nice way of defining your DAGS because it allows you to scale it to any number of tasks depending on any number of tasks with still just two lines.

user_22774450
Related