Basic DAG Issues - Bash Command Runs in Terminal and Not Airflow

Clyde 0
Viewed 16

I'm trying to run this DAG in Airflow with the intention of automating some python scripts. To test it I'm running a super simple .py as seen below.

When running 'python3 /home/j/git/J/Code_Modules/TEST.py' in my linux terminal in the Astro_Projects directory it works just fine. Won't seem to run in the DAG though.

Log file is returning "No such file or directory." The .py script is in my "Code_Modules" folder and the DAG is in my "Astro_Projects" folder as shown here:

enter image description here

I'm pretty new to this so any help is greatly appreciated. Thanks

#%%
import os
from datetime import datetime, timedelta
from airflow.models import DAG, Variable
from airflow.operators.bash_operator import BashOperator

# DAG Settings
args = {
    'owner': 'J',
    'start_date': datetime(2022, 9, 14),
    'retries': 1,
    'retry_delay': timedelta(minutes=1),
    'dagrun_timeout': timedelta(minutes=15),
    'email_on_failure': True,
    'email_on_retry': False,
    'catchup': False
}

dag = DAG(
    dag_id='TEST_DAG',
    default_args=args,
    schedule_interval='0 6 * * *',
    tags=['Prod','TEST']
)

send_reports = BashOperator(
    task_id='TEST',
    bash_command='python3 /home/j/git/J/Code_Modules/TEST.py',
    dag=dag
)

# %%

"TEST.py" is just this:

import os
print('thisworked')
airflow python
1 Answers

use relative path:

send_reports = BashOperator(
    task_id='TEST',
    bash_command='python3 ../Code_Modules/TEST.py',
    dag=dag
)

but it makes more sense to use PythonOperator to execute python functions,

eshirvana0
Related