Airflow EC2 machine not showing correct location with os.getcwd() as /opt/***/requirements.txt

userUHx21 0
Viewed 34

I want to access requirements.txt using the BashOperator in the airflow dag on a ec2 machine

Folder structure is like :- 
airflow
  |dags Folder
  |requirements.txt
  |other files

currently I am running the dag on airflow EC2 machine this dag basically is used to run a command which installs newly added pip modules in EC2 machine from requirements.txt when I am running the bash command with pip install -r {location}/requirements.txt location = os.getcwd() -> gives current directory But here it is giving something like /opt/***/requirements.txt that is why it is giving no such file directory there any solution for this I am using apache airflow 2.3.4 version and python 3.10 version

Code as follows:

import os
from airflow import DAG
from datetime import datetime
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator

default_args = {"owner":"airflow","start_date":datetime(2022,1,1)}

location =os.getcwd()

#Clear The contents of Requirements File
def remove_modules_from_file():
    with open(f'{location}/requirements.txt', 'r+') as f:
        f.truncate(0)
    

with DAG(dag_id="new_modules_install",default_args=default_args,schedule_interval='@daily') as dag:
    install_module = BashOperator(
        task_id='install_module',
        bash_command=f'pip install -r {location}/requirements.txt'
    )
    remove_modules_from_file = PythonOperator(
        task_id='remove_modules_from_file',
        python_callable=remove_modules_from_file
    )

    install_module >> remove_modules_from_file

Error message in Dag console

[2022-09-06, 11:04:05 UTC] {subprocess.py:62} INFO - Tmp dir root location: / tmp
[2022-09-06, 11:04:05 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'pip install -r/opt/***/requirements.txt'l
[2022-09-06, 11:04:05 UTC] {subprocess.py:85} INFO - Output:
[2022-09-06, 11:04:06 UTC] {subprocess.py: 92} INFO - Defaulting to user installation because normal site-packages is not writeable
[2022-09-06, 11:04:06 UTC] {subprocess.py: 92} INFO - ERROR: Could not open requirements file: [Errno 2] No such file or directory: '/opt/***/requirements.txt'
[2022-09-06, 11:04:06 UTC] {subprocess.py:92} INFO
[2022-09-06, 11:04:06 UTC] {subprocess.py: 92} INFO - [notice] A new release of pip available: 22.1.2 -> 22.2.2
[2022-09-06, 11:04:06 UTC] {subprocess.py: 92} INFO - [notice] To update, run: python -m pip install -upgrade pip
[2022-09-06, 11:04:07 UTC] {subprocess.py: 96} INFO • Command exited with return code 1
[2022-09-06, 11:04:07 UTC] (taskinstance.py: 1909} ERROR - Task failed with exception
airflow python
1 Answers

The ['bash', '-c', 'pip install -r/opt/***/requirements.txt' is not what you think it is. This is "secret masking" kicking in. You likely have "airflow" as a password set in connection and airflow will automatically replace all "passwords" with "***" - you can test it out by changing passwords for all your connections to be something different than "airflow".

So likely your command is right - but the file is indeed missing at the machine/container that the worker executes on. You have to be aware, that the command is executed in Worker not in scheduler or webserver. Which means that your worker might not have access to the file even if your scheduler has. So you need to check if the file is present in your worker (and where your worker is, depends on the executor you use.

Another problem you attempt to run different tasks with the assumption that they will run on the same machine. While this might be working when you use LocalExecutor and single machine, this will break when you distribute the code and you will have multipler workers or K8S executor and each task might be executed on a different machine - so mixing .get_cwd() and cleaning/installing the requirements in one DAG, does not make them execute on the same machine.

Another problem is that you are deleting the contents after using them and then if you re-run the dag (assuming you are using LocalExecutor) - you will get empty file.

Yet another problem is that you are mixing method name remove_modules_from_file with task name remove_modules_from_file - one of them is callable, and one of them is task. I am not sure what happens in this case to be honest - I'd rather avoid reusing the same name for two different things.

I think there are plenty of things to improve there and I'd advice you to take more look at the examples and probably look at some online courses of Airlfow - I recommend https://academy.astronomer.io/ - you seem to have a bit more things to grasp than just a missing file.

BTW. This is actually quite bad practice to have your passwords set as "airflow" so I hope this is just a development server of yours not a production one.

Jarek Potiuk0
Related