How to import DAG with specific Parameters

user_8209819 0
Viewed 65

I have a DAG that I want to launch in Composer with some parameters. I can't use Airflow Global Variables as I want to launch the same DAG with different "Context". My Parameters representing this context

Thanks for the help

google-cloud-composer
1 Answers

What you can do is to create a dag as a function on a separate file and then just reuse you dag code with different dag configurations. Below represents is the approach in pseudo code:

Dag Folder

#ie:
#machine_export.py
#car_export.py
#plane_export.py

from logic import export_pub_template

config = {
'param1': ...
'param2': ...
}

with DAG(dag_id='',
         ...
     ) as dag:

   export_pub_template(config)

Logic Folder

#push_data_template_dag.py 

def export_pub_template(config):

  get_data = get_data_custom_operator(config.params)
  export_data = send_data_to_db(config.params)

  get_data >> export_data

Also, here is also a similar approach. It follows the same pattern which is to break up the DAG code so you can reuse it in similar DAGs.

For details about writing DAGs you can check this link.

user_145515670
Related