XCOM is a tuple, how to pass the right value to two different downstream tasks

user_9291262 0
Viewed 71

I have an upstream extract task, that extracts files into two different s3 paths. This operator returns a tuple of the two separate s3 paths as XCOM. How do I pass the appropriate XCOM value to the appropriate task?

extract_task >> load_task_0 load_task_1

airflow
1 Answers

Probably a little late to the party, but will answer anyways.

With TaskFlow API in Airflow 2.0 you can do something like this using decorators:

@task(multiple_outputs=True)
def extract_task():
    return {
        "path_0": "s3://path0",
        "path_1": "s3://path1",
    }

Then in your DAG:

@dag()
def my_dag():
    output = extract_task()
    load_task_0(output["path_0"])
    load_task_1(output["path_1"])

This works with dictionary, probably won't work with tuple but you can try.

user_145757110
Related