Install python dependencies for apache airflow

Question

I am using apache airflow for running my dags. I want to install the python dependencies: requests==2.22.0

My docker compose file for webserver, scheduler and postgres is:

version: "2.1"
services:
  postgres_airflow:
    image: postgres:12
    environment:
        - POSTGRES_USER=airflow
        - POSTGRES_PASSWORD=airflow
        - POSTGRES_DB=airflow
    ports:
        - "5432:5432"

  postgres_Service:
    image: postgres:12
    environment:
        - POSTGRES_USER=developer
        - POSTGRES_PASSWORD=secret
        - POSTGRES_DB=service_db
    ports:
        - "5433:5432"
 
  scheduler:
    image: apache/airflow
    restart: always
    depends_on:
      - postgres_airflow
      - postgres_Service
      - webserver
    env_file:
      - .env
    volumes:
        - ./dags:/opt/airflow/dags
    command: scheduler
    healthcheck:
        test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
        interval: 30s
        timeout: 30s
        retries: 3

  webserver:
    image: apache/airflow
    restart: always
    depends_on:
        - pg_airflow
        - pg_metadata
        - tenants-registry-api
        - metadata-api
    env_file:
      - .env
    volumes:
        - ./dags:/opt/airflow/dags
        - ./scripts:/opt/airflow/scripts
    ports:
        - "8080:8080"
    entrypoint: ./scripts/airflow-entrypoint.sh
    healthcheck:
        test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
        interval: 30s
        timeout: 30s
        retries: 3

My dag file is:

import requests
from datetime import datetime

from airflow import DAG

from airflow.operators.python_operator import PythonOperator

default_args = {'owner': 'airflow',
                'start_date': datetime(2018, 1, 1)
                }

dag = DAG('download2',
          schedule_interval='0 * * * *',
          default_args=default_args,
          catchup=False)


def hello_world_py():
    requests.post(url)
    print('Hello World')


with dag:
    t1 = PythonOperator(
        task_id='download2',
        python_callable=hello_world_py,
        requirements=['requests==2.22.0'],
        provide_context=True,
        dag=dag
    )

The problems I am facing are:

I can not use PythonVirtualenvOperator to install the requirements as I am facing an issue Airflow log file exception
I can not use something like:

    build:
      args:
        PYTHON_DEPS: "requests==2.22.0"

as I do not have Dockerfile in the context. I have image with apavhe/airflow.

I can not use volume mount ./requirements.txt:requirements.txt in initdb as I am not using the initdb container. I am using just the command in script airflow initdb.

Any solution to the above three problems would work.

Install python dependencies for apache airflow

0 Answers