How to parse nested macros in Airflow

Viewed 382

I'm currently facing a challenge in terms of parsing nested macros. Below is my DAG File

from airflow import DAG
from airflow.utils.dates import days_ago
from datetime import timedelta
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.models import Variable
from apty.utils.date import date_ref_now

default_args = {
    "owner": "Akhil",
    "depends_on_past": False,
    "start_date": days_ago(0),
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 0,
    "retry_delay": timedelta(minutes=5),
}

dag = DAG(
    "user_sample",
    default_args=default_args,
    description="test",
    schedule_interval=None,
    catchup=False,
)



def sample_app(hello=None):
    return hello

extra_attrs = {"date_stamp":"{{ds}}",
"foo":"bar"}

start = DummyOperator(task_id="start", dag=dag)


python = PythonOperator(
    python_callable=sample_app,
    task_id="mid",
    dag=dag,
    params={"date_stamp": extra_attrs["date_stamp"]},
    op_kwargs={"hello": "{{params.date_stamp}}"},
)

start >> python

I have a scenario where I need to pass {{ds}} as one of the parameters to my operator, after which I'll use that parameter as my wish either passing as an op_kwargs / op_args. (I have used Python Operator as an example but I would be using my own custom Operator).

Here I would like to make it clear that {{ds}} is passed as a parameter value only, I don't want it to be written anywhere i.e in op_kwargs as per this example.

When I try to run it I'm getting return value from python callable as {{ds}} but not the current date_stamp.

Please help me out.

1 Answers

Template or macro variables are only available for parameters that are specified as template_fields on the operator class in use. This depends on the specific version and implementation of Airflow you're using, but here's the latest https://github.com/apache/airflow/blob/98896e4e327f256fd04087a49a13e16a246022c9/airflow/operators/python.py#L72 for the PythonOperator. Since, as you say, you control the operator in question, you can specify any fields you want on the class definition's template_fields. (This all assumes your class inherits from BaseOperator.)

Related