Pulling Airflow xcom outside of an operator?

Viewed 210

I have some dynamicism in my DAG and I pull these XCOM file paths within each successive task:

def _thing(task_id, num_executors):
    ti = context['ti']
    task_to_pull_from = 'task_name_{}_{}'.format(task_name, index)
    pulled_dynamo_data = ti.xcom_pull(key='return_value', task_ids=task_to_pull_from)

    paths_I_want = [
        "--target-path", "s3://{}{}".format("", ""),
        "--source-path", "s3://{}{}".format(bucket_name, "")
...

_thing is just a Python function in this case, which I invoke, however, I want to pull XCOM values within this function. Is it possible to have a PythonOperator that calls a function which returns an operator for me? In this case, this returns a Spark Livy operator.

0 Answers
Related