passing parameters to the UI when triggering a dag on airflow

user_15139548 0
Viewed 3097

I'm trying to find a way to generalize passing parameters of a dag when triggering it from the UI itself,I know I should pass it as a key/value pair format but I don't know how to parse those parameters in the dag script itself

for example if I pass the conf pair {"dir_of_project":"root/home/project"}

what should i do inside the dag script so that I have inside it a variable called dir_of_project that equals the path above so that I can use it further in my code. I tried to do the following inside the script but obviously i'm doing it wrong :

dir of project = "{{ dag_run.conf['file_name'] }}"

for example let's say i did this:

def func(**kwargs):
    file_name = kwargs['dag_run'].conf.get('dir_of_project')
    return file_name
op = PythonOperator(
    task_id='task',
    python_callable=func,
    dag=dag,
    provide_context=True
)


sampling_out_parent_dir="/root/home/myfolder/sampled-vids"
file_path=os.path.join(sampling_out_parent_dir,file_name)

this would result in an undef. variable error as it's defined only in the scope of the fn and the operator and i can't used it outside like this, so how can i make it work?

airflow python
2 Answers

I believe your issue is because you are using Jinja somewhere that isn't being templated.

For input of {"dir_of_project":"root/home/project"} when you manually trigger DAG in the UI or executing with CLI:

airflow trigger_dag your_dag_id --conf '{"dir_of_project":"root/home/project"}'

you can extract with:

{{ dag_run.conf['dir_of_project'] }}

or if you use PythonOperator with:

kwargs['dag_run'].conf['dir_of_project']

Example:

def func(**kwargs):
    file_name = kwargs['dag_run'].conf.get('dir_of_project')
    # Rest of the code

op = PythonOperator(
    task_id='task',
    python_callable=func,
    dag=dag,
    provide_context=True # Remove this if you are using Airflow >=2.0
)
user_146244090

I wanted to updated this as I solved the issue a while back but forgot to, after much research I found out that's actually impossible to extract variables with their values from airflow's operator and use them outside of it, however I found a workaround this by using template fields, airflow uses something called Xcom variables which variables that can be pushed and pulled from and to operators( basically a comm. system between them), it pretty much fets the job done if you want to extract the variable from one operator to another. further more if you want to build new normal variables depending on an Xcom one, you can do that by using the Xcom one as a template filed and even build further ones, as long as you'll eventually use those normal ones inside an operator which then will be able to render that template. for example: if I push an Xcom variable in one of my tasks operator and that variable has an integer value = 100, i can get the variable by using this: newvar="{{ task_instance.xcom_pull(task_ids='%s') }}"%task_id where task_id is the name of the task you pushed the variable in, further more i can use newvar outside the operator, it will carry around that value as a string until again passed to an operator which will render the template field and replace it with the original value 100. for more info refer to airflow Xcom documentation :here

user_151395480
Related