AIRFLOW - Setting relationships from a list of operators

user_9291262 0
Viewed 1319

I have a list of operators that were appended to a list in a for loop. I would like to set a relationship for my DAG where each task is set downstream in the order of the list. For example

task_list = []
for operator in operator_array:
    task = operator
    task_list.append(task)

# theoretically I would want something like this:

some_other_operator >> task_list[0] >> task_list[1] >> task_list[2] and so on and so forth for however long the length of the list is

Thank you

airflow list python
2 Answers

There's a helper function created for that, will be much more performant than the accepted answer

See airflow.utils.helpers module

from airflow.utils.helpers import chain

chain(*task_list)
user_115218540

For a list of tasks named task_list, you can do:

some_other_operator >> task_list[0]

for i in range(len(task_list) - 1):
    task_list[i] >> task_list[i + 1]
user_124002140
Related