How to fix "permission denied" error while running docker-compose

user_10617728 0
Viewed 2631

I'm running Airflow 2.0 using docker, when i run sudo docker-compose up --build I'm getting an error

ERROR: for apache_airflow_scheduler_1 Cannot start service scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "./scripts/airflow-entrypoint.sh": permission denied: unknown

ERROR: for webserver Cannot start service webserver: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "./scripts/airflow-entrypoint.sh": permission denied: unknown

ERROR: for scheduler Cannot start service scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "./scripts/airflow-entrypoint.sh": permission denied: unknown

ERROR: Encountered errors while bringing up the project.

my layout for docker-compose is

version: "2.1"
services:
  postgres: # metadata database
    image: postgres:12
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
      - "5434:5432" # mapping the port 5432 of postgres container with 5434 of local machine

  scheduler:
    build:
      context: .
      dockerfile: Dockerfile
    restart: on-failure
    command: scheduler
    entrypoint: ./scripts/airflow-entrypoint.sh
    depends_on:
      - postgres
    env_file:
      - .env
    ports:
      - "8794:8793"
    volumes:
      - ./dags:/opt/airflow/dags
      - ./airflow-logs:/opt/airflow/logs
      - ./scripts:/opt/airflow/scripts
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

  webserver:
    build:
      context: .
      dockerfile: Dockerfile
    hostname: webserver
    restart: always
    depends_on:
      - postgres
    command: webserver
    env_file:
      - .env
    volumes:
      - ./dags:/opt/airflow/dags
      - ./scripts:/opt/airflow/scripts
      - ./airflow-logs:/opt/airflow/logs
    ports:
      - "8088:8080"
    entrypoint: ./scripts/airflow-entrypoint.sh
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 32

Dockerfile

FROM apache/airflow
USER root
ARG AIRFLOW_HOME=/opt/airflow
ADD dags /opt/airflow/dags
RUN pip install --upgrade pip
RUN chown -R airflow:airflow $AIRFLOW_HOME
USER airflow
RUN pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org boto3
USER ${AIRFLOW_UID}

My layout for my airflow-entrypoint.sh is

#!/usr/bin/env bash
airflow db init
airflow db upgrade
airflow users create -r Admin -u admin -e MY_EMAIL -f MY_FIRSTNAME -l MY_LASTNAME -p admin
airflow scheduler &
airflow webserver

How can i resolve this? I assume i execute chmod +x but where do I place it since the two images webserver and scheduler are custom images.

airflow-scheduler airflow docker-compose docker
1 Answers

I do not think your problem is permission, your problem is that you are not using docker-compose the way it is intended to be used with Airflow.

First of all - do not use sudo - this runs your docker compose as root and you do not need to do that.

Just follow https://docs.docker.com/engine/install/linux-postinstall/ and make it possible to run docker as your own user.

Secondly, You are trying to run your scheduler and webserver in the same container, which makes very little sense - especially that you already have one container for scheduler and one for webserver and yet for some reason you try to start both scheduler and webserver in your entrypoint? This way you end up with both scheduler and webserver running twice in both containers.

Generally one container should run one process (either webserver or scheduler) and you do not need custom entrypoint to start those.

Thirdly - running db init and db upgrade in the same entrypoint makes very little sense as well, they will be run every time you start any command and this is entirely not needed. Since you are using a postgres db, it's enough that you run it once using a custom "init" job.

If you get rid of those, you will not need custom entrypoint (and if you are using the official Airflow image, you should not do it anyway, because the official image of the entrypoint already handles a lot of stuff for you: https://airflow.apache.org/docs/docker-stack/entrypoint.html and skipping the entrypoint or replacing it with your own is pretty bad idea.

You can instead use "bash" command of the image and run whatever you want there.

I'd heavily recommend you take a look at the quick-start docker-compose we have in Airflow as an inspiration of what and how you shoudl do with the docker-compose : https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

user_5167010
Related