I have a docker-compose.yml with:
version: "3"
services:
postgres:
image: "postgres:9.6"
container_name: "postgres"
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- "5432:5432"
volumes:
- ./data/postgres:/var/lib/postgresql/data comment initdb after you will have use it at first run
initdb:
build: .
entrypoint: airflow initdb
depends_on:
- postgres
webserver:
container_name: my-container
build: .
restart: always
depends_on:
- postgres
volumes:
- ./dags:/usr/local/airflow/dags
ports:
- "8080:8080"
entrypoint: airflow webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
scheduler:
build: .
restart: always
depends_on:
- postgres
volumes:
- ./dags:/usr/local/airflow/dags
entrypoint: airflow scheduler
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-scheduler.pid ]"]
interval: 30s
timeout: 30s
retries: 3
A dockerfile with just:
FROM puckel/docker-airflow:1.10.6
RUN pip install --user psycopg2-binary
ENV AIRFLOW_HOME=/usr/local/airflow
COPY ./config/airflow.cfg /usr/local/airflow/airflow.cfg
The cfg file is the one that you cand find here, but I also tried with this one.
I run in different terminals and as a sequence:
docker-compose up postgres
docker-compose up initdb
The first one work well. But the second one returns:
initdb_1 | /usr/local/lib/python2.7/dist-packages/airflow/configuration.py:241: FutureWarning: The task_runner setting in [core] has the old default value of u'BashTaskRunner'. This value has been changed to u'StandardTaskRunner' in the running config, but please update your config before Apache Airflow 2.0.
initdb_1 | FutureWarning
initdb_1 | [2020-03-12 16:10:07,456] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
initdb_1 | Traceback (most recent call last):
initdb_1 | File "/usr/local/bin/airflow", line 25, in <module>
initdb_1 | from airflow.configuration import conf
initdb_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 47, in <module>
initdb_1 | settings.initialize()
initdb_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 377, in initialize
initdb_1 | configure_orm()
initdb_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 266, in configure_orm
initdb_1 | engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
initdb_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
initdb_1 | return strategy.create(*args, **kwargs)
initdb_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 160, in create
initdb_1 | engineclass.__name__))
initdb_1 | TypeError: Invalid argument(s) 'pool_pre_ping' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine. Please check that the keyword arguments are appropriate for this combination of components.
This is the minimal expression of the docer that I am trying to do. I added parameters to cfg to fix it, but nothing works.
Any idea?