where should be place the python script to run it from Airflow dag?

user_11903813 0
Viewed 126

I am trying to run a simple dag in Airflow to execute a python file and it is throwing the error can't open the file '/User/....'.

below is the script I am using.

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime,timedelta

default_args = {
   'owner': 'airflow',
   'start_date': datetime(2021,3,2),
   'depends_on_past': False,
   'retries': 0
}

dag=DAG(dag_id='DummyDag',default_args=default_args,catchup=False,schedule_interval='@once')

command_load='python /usr/local/airflow/dags/dummy.py '

#load=BashOperator(task_id='loadfile',bash_command='python /Users/<user-id>/MyDrive/DataEngineeringAssignment/dummydata.py',dag=dag)


start=DummyOperator(task_id='Start',dag=dag)
end=DummyOperator(task_id='End',dag=dag)

dir >> start >> end

Is there anywhere I am going wrong?

airflow python
1 Answers

Option 1:

The file location (dummydata.py) is relative to the directory containing the pipeline file (DAG file).

dag=DAG(
    dag_id='DummyDag',
    ...
)

load=BashOperator(task_id='loadfile',
                  bash_command='python dummydata.py',
                  dag=dag
                  )

Option 2: define your template_searchpath as pointing to any folder locations in the DAG constructor call.

dag=DAG(
    dag_id='DummyDag',
    ...,
    template_searchpath=['/Users/<user-id>/MyDrive/DataEngineeringAssignment/']
)

load=BashOperator(task_id='loadfile',
                  # "dummydata.py" is a file under "/Users/<user-id>/MyDrive/DataEngineeringAssignment/"
                  bash_command='python dummydata.py ', # Note: Space is important!
                  dag=dag
                  )

For more information you can read about it in the docs

user_146244090
Related