FileNotFoundError: [Errno 2] No such file or directory when a task tried to save a file

Viewed 1689

I'm trying to copy the content of a csv file into postgres database , in two tasks , the first task downloads the csv file and saves it in /temp folder and the other one is a postgres task that copies the elements into the database. However the task fails to save the file with a filenotfound error when trying to save it outside of the dag folder.

The callable function that saves the file :

def get_events(country_code,start_date, max_days = 365):
    AIRFLOW_HOME = os.getenv('AIRFLOW_HOME')
    iterate_date = start_date
    has_events = False
    for i in range(0, max_days):
        try:
            results = get_event_perday_bycountry(start_date, country_code)
            print(iterate_date)
            if not results.empty:
                results.to_csv('/mnt/c/Users/heyas/project/STOCK/temp/data.csv' ,index = False, mode = "a", header = False)
                has_events = True
        except ValueError as err:
            print(err)
        iterate_date -= timedelta(days=1)
    if has_events:
        print("events Pulled")
    else:
        print("No events obtained.")

I used AIRFLOW_HOME variable to save a file in the dag folder before but my postgres can't copy the content of the csv file from there .

docker-compose file for AIRFLOW+ Postgres :

version: '3.7'
services:
    postgres:
        image: postgres:9.6
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow
        logging:
            options:
                max-size: 10m
                max-file: "3"
        volumes:
            - ./setup/raw_input_data:/data
            - ./temp:/temp
        ports:
            - "5432:5432"
    webserver:
        image: puckel/docker-airflow:1.10.9
        restart: always
        depends_on:
            - postgres
        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - AIRFLOW_CONN_POSTGRES_DEFAULT=postgres://airflow:airflow@postgres:5432/airflow
            - FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
        logging:
            options:
                max-size: 10m
                max-file: "3"
        volumes:
            - ./dags:/usr/local/airflow/dags
            - ./temp:/temp
            # - ./plugins:/usr/local/airflow/plugins
        ports:
            - "8080:8080"
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3

I tried to work around it by saving a file directly in /temp folder , my directory :

├── Postgre
├── __pycache__
├── dags
│   ├── __pycache__
│   ├── data
│   ├── gdelt
│   │   └── __pycache__
│   ├── gdelt-0.1.10.dist-info
│   └── scripts
├── setup
│   ├── raw_input_data
│   └── temp
└── temp

How can I save the file in the temp folder , or put a common folder between Postgres and AIRFLOW in my directory?

1 Answers
Related