Apache Airflow How to xcom_pull() value into a DAG?

user_7813928 0
Viewed 7706

I have a custom operator which pushes XCOM value as below:

...
task_instance = context['task_instance']
task_instance.xcom_push("list_of_files",file_list)
...

It works fine. I have a dag definition file (my_dag.py) where I create a task by using my own operator, it pushes the XCOM value then I want to do for in loop by using this xcom value. How to pull it?

airflow directed-acyclic-graphs python
2 Answers

It is generally bad practice to access xcom from the dag itself rather than from a task in the dag. That said, sometimes it is necessary. For example, you may need to do this when dynamically creating dags.

Here is an example of me pulling some unrun jobs within a dag. I'm using this in the context of a subdag, so I can rest assured that the xcom will always contain the information assuming the method is running.

    xcom_unrun_jobs = None
    if len(parent_dag.get_active_runs()) > 0:
        tis = parent_dag.get_task_instances(settings.Session, start_date=parent_dag.get_active_runs()[-1])[-1]
        xcom_unrun_jobs = tis.xcom_pull(dag_id=parent_dag._dag_id, task_ids=unrun_job_task_id)
user_18129930
Related