I'm trying to copy the content of a csv file into postgres database , in two tasks , the first task downloads the csv file and saves it in /temp folder and the other one is a postgres task that copies the elements into the database. However the task fails to save the file with a filenotfound error when trying to save it outside of the dag folder.
The callable function that saves the file :
def get_events(country_code,start_date, max_days = 365):
AIRFLOW_HOME = os.getenv('AIRFLOW_HOME')
iterate_date = start_date
has_events = False
for i in range(0, max_days):
try:
results = get_event_perday_bycountry(start_date, country_code)
print(iterate_date)
if not results.empty:
results.to_csv('/mnt/c/Users/heyas/project/STOCK/temp/data.csv' ,index = False, mode = "a", header = False)
has_events = True
except ValueError as err:
print(err)
iterate_date -= timedelta(days=1)
if has_events:
print("events Pulled")
else:
print("No events obtained.")
I used AIRFLOW_HOME variable to save a file in the dag folder before but my postgres can't copy the content of the csv file from there .
docker-compose file for AIRFLOW+ Postgres :
version: '3.7'
services:
postgres:
image: postgres:9.6
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
logging:
options:
max-size: 10m
max-file: "3"
volumes:
- ./setup/raw_input_data:/data
- ./temp:/temp
ports:
- "5432:5432"
webserver:
image: puckel/docker-airflow:1.10.9
restart: always
depends_on:
- postgres
environment:
- LOAD_EX=n
- EXECUTOR=Local
- AIRFLOW_CONN_POSTGRES_DEFAULT=postgres://airflow:airflow@postgres:5432/airflow
- FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
logging:
options:
max-size: 10m
max-file: "3"
volumes:
- ./dags:/usr/local/airflow/dags
- ./temp:/temp
# - ./plugins:/usr/local/airflow/plugins
ports:
- "8080:8080"
command: webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
I tried to work around it by saving a file directly in /temp folder , my directory :
├── Postgre
├── __pycache__
├── dags
│ ├── __pycache__
│ ├── data
│ ├── gdelt
│ │ └── __pycache__
│ ├── gdelt-0.1.10.dist-info
│ └── scripts
├── setup
│ ├── raw_input_data
│ └── temp
└── temp
How can I save the file in the temp folder , or put a common folder between Postgres and AIRFLOW in my directory?