Create airflow dag within django rest api views

Viewed 268

Currently am learning airflow to perform my batch processing task in python.

I just created a python file under dag folder in AIRFLOW_HOME path that exist in root directory of my Django rest api project. I tested by creating some sample dags in file that exist in dags folder.

But i want to create airflow dag when one of my Django rest api is called. Basically am getting job metadata from UI and storing it in my database. So i want to create a dag using that data in my Django views file.

I tried by pasting same code in my Django views file that i used in files under dags folder but its not creating dags. Here is what i tried in my Django views file

dag_id = 'test_dag_1'

default_args = {'owner': 'airflow',
      'start_date': datetime.datetime(2021, 9, 13)
     }

schedule = '@daily'
dag_number = 1
dag = DAG(dag_id,
   schedule_interval=schedule,
   default_args=default_args)

with dag:
    t1 = PythonOperator(
        task_id='hello_world',
        python_callable=hello_world_py
    )

    globals()[dag_id] = dag

But this did not worked. I also tried by changing dags_folder value in airflow.cfg to my django views.py file

I also check airflow stable rest api but its doe snot support to create a new dag.

Can anyone please guide on how can i create dags in my Django views.py when my api is being called

0 Answers
Related