Airflow xcom return a string in list format instead of just a string value?

user_10281199 0
Viewed 1290

I have an airflow operator that returns a string value and the task is named 'task1'. So after execution I go into xcom and check the return_value and its just a string (screenshot below).

xcom key/value screenshot

Then I have an operator that follows named task2 takes an input from the xcom value from task1 like below:

"{{ ti.xcom_pull(task_ids=['task1'],key='return_value')}}"

The problem is the value it gets is a list converted to a string.

Value in xcom: this is just a string

Value returned by xcom pull (jinga template version): ['this is just a string']

So is there a way I can update xcom pull (jinga version) shown above to just pull the value? I don't have access inside the operator its being passed into or I could put some logic to convert string to a list then get the value only (but that would not be ideal and is not an option anyway).

Also, I think its work mentioning I tried to do something similar but using the Python operator then doing an xcom pull using the content inside the python code and the value was returned just fine. So I'm not sure why the xcom pull using the Jinja Templating does this and how I can get around this? I'm hoping there is something I can do that I'm not aware of to easily get the output I want. The python operator code that works is below (just as an fyi...)

def python_code_task3(**context): 
value = context['ti'].xcom_pull(task_ids='task1', key='return_value') 
logging.info("Value: " + value)

And this code just outputs the value like I want this is just a string

I really just want to use the jinga template version and have it retrieve and pass in the string. Not a string representation of a list with the string value as one item in the list.

airflow
1 Answers

There is a slight difference between the two ways you are pulling the XCom in your code snippets: one has task_ids=["task_1"] (a list arg) while the other has task_ids="task_1" (a str arg).

The argument type of task_ids matters when using xcom_pull(). Airflow will infer that if you pass a list of task IDs, there should be multiple tasks to pull XComs from and will return a list containing all of the retrieved XComs. Otherwise if the type is simply a string aka a single task ID, a single XCom value is returned. Here is a link to the code where this is done.

It's also worth noting that Jinja-templated values are rendered as strings by default. However, with Airflow 2.1 you can set a parameter called render_template_as_native_obj to True at the DAG level. This will now render Jinja-templated values as native Python objects (list, dict, etc.) when applicable. More info on this concept here.

user_157723770
Related