Airflow BashOperator - cannot create directory

Viewed 302

I have a 'create_dir.sh' script in directory: \wsl.localhost\Ubuntu\home\username\dataeng\airflow\dags\bash_operator\scripts

#!/bin/bash
cd /home/username/dataeng/
mkdir test_dir

DAG file 'bash_operator_create_directory.py' is the following:

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "start_date": datetime(2022, 4, 3),
    "email": ["xxx.xxx@gmail.com"],
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5)
}

dag = DAG("bash_operator_create_directory", default_args=default_args, schedule_interval=timedelta(1))
task = BashOperator(task_id="create_directory", bash_command="/home/username/dataeng/airflow/dags/bash_operator/scripts/create_dir.sh ", dag=dag)

After running this task, it's giving the following error message:

[2022-04-03, 08:48:38 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', '/home/username/dataeng/***/dags/bash_operator/scripts/create_dir.sh ']
[2022-04-03, 08:48:38 UTC] {subprocess.py:85} INFO - Output:
[2022-04-03, 08:48:38 UTC] {subprocess.py:89} INFO - bash: /home/username/dataeng/***/dags/bash_operator/scripts/create_dir.sh: No such file or directory
[2022-04-03, 08:48:38 UTC] {subprocess.py:93} INFO - Command exited with return code 127

Then I modified BashOperator task to run mkdir command directly:

task = BashOperator(task_id="create_directory", bash_command="mkdir /home/username/dataeng/test_dir", dag=dag)

After running this task, it's giving the following error message:

[2022-04-03, 07:51:04 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'mkdir /home/username/dataeng/test_dir']
[2022-04-03, 07:51:04 UTC] {subprocess.py:85} INFO - Output:
[2022-04-03, 07:51:04 UTC] {subprocess.py:89} INFO - mkdir: cannot create directory ‘/home/username/dataeng/test_dir’: No such file or directory
[2022-04-03, 07:51:04 UTC] {subprocess.py:93} INFO - Command exited with return code 1

Any idea what's wrong?

1 Answers

you should try this:

  1. put your bash command in a .sh file
  2. elevate .sh file permissions
  3. run that bash file instead of the direct bash command

I know you already have a file but created a list for future references. your problem is most likely the file's permission which you should take care of with chmod

Related