New libraries required Airflow

user_14245919 0
Viewed 2175

I am currently using docker-compose up to run my apache airflow image which i get from docker. However, when I try to run my pythonOperator(python script which contain additional libraries), errors occur when building the airflow.

My docker-compose.yml:

version: '3'
services:
  postgres:
    image: postgres:9.6
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
      - "5432:5432"

  webserver:
    image: puckel/docker-airflow:1.10.1
    build:
      context: https://github.com/puckel/docker-airflow.git#1.10.1
      dockerfile: Dockerfile
      args:
        AIRFLOW_DEPS: gcp_api
        PYTHON_DEPS: sqlalchemy==1.2.0
    restart: always
    depends_on:
      - postgres
    environment:
      - LOAD_EX=n
      - EXECUTOR=Local
      - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo=
    volumes:
      - /dags:/usr/local/airflow/dags # My dags files
      - /src:/usr/local/airflow/src # my src files and its the one that contain the new libraries
      # Uncomment to include custom plugins
      # - ./plugins:/usr/local/airflow/plugins
    ports:
      - "8085:8080"
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

The command i use when running docker:

docker-compose up -d

The error messsage I received.

image

Could anyone tell me how can I fix this error? How should I install new packages when I want to add more pythonOperators?

docker-compose docker python
1 Answers

You can install in the docker your required libraries, that is quick solution:

Access the container id of airflow worker as root user

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                 PORTS                                        NAMES
c73425b30981        puckel/docker-airflow:1.10.9   "/entrypoint.sh work…"   7 hours ago         Up 7 hours             5555/tcp, 8080/tcp, 8793/tcp                 airflow_worker_1

docker exec -it --user root c73 /bin/bash

Run apt get update root@c73425b30981:/usr/local/airflow# apt-get update

Then install your python libraries root@c73425b30981:/usr/local/airflow# pip install Janome

Then you will be able to use janome library

user_107389300
Related