How can I run jupyter in airflow with docker-compose?

user_13071391 0
Viewed 554

i'd like to slip in a jupyter server on my dockerized airflow so that i can develop more easily in that environment. i see from Using Docker-Compose, how to execute multiple commands. that you can easily run multiple commands in docker-compose with a 'bash -c' .

here is what i've tried:

command: bash -c "airflow webserver ; nohup jupyter notebook --ip 0.0.0.0  --no-browser &"

also tried:

command: bash -c "airflow webserver && nohup jupyter notebook --ip 0.0.0.0  --no-browser &"

i know this is possible because i can do something like:

docker exec -it -u airflow  8b2 jupyter notebook --ip 0.0.0.0  --NotebookApp.token='airflow' --no-browser

this of course required adding the port 8888:8888 mapping for j:

ports:
  - ${AIRFLOW_WEBSERVER_PORT:-8080}:8080
  - 8888:8888

thanks!

jupyter airflow docker-compose
2 Answers

deploying more than one application inside a container is not a best practice. each container should do one thing and do it well. reference

You can do the following:

  1. Separate Airflow and Jupyter into separate containers.
  2. Use a Docker network to connect the two.

Let me show you this project that satisfies most of your needs and works well for me. Have a look at it. You might be inspired by it.

user_15955530

As @Numb95 points out, the problem is best solved by specifying a dedicated jupyter container.

The 'quick-start' docker-compose at this time is: https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml.

Using the quick-start docker-compose above as a guid, the code below can be placed in docker-compose.yaml, but would need the airflow image specified within to be custom compiled with jupyter-core and jupyter:

  airflow-jupyter:
    <<: *airflow-common
    #image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.2}
    image: <CUSTOM_IMAGE_DESCRIBED_ABOVE>
    container_name: airflow_jupyter
    command: bash -cx "jupyter notebook --ip 0.0.0.0  --NotebookApp.token='airflow' --no-browser"
    ports:
      - 8888:8888
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

To install the jupyter packages, the DockerFile would need to be rebuilt to include:

RUN pip install -U jupyter-core --user
RUN pip install -U jupyter --user
RUN chmod -R 775 /home/airflow/.local/share/jupyter
user_130713910
Related