Notebook based on jupyter/all-spark-notebook docker image not picking up custom python version

user_6701520 0
Viewed 1067

Summary

I am trying to execute this simple python code snippet in an all-spark-notebook which is supposed to be executed in a local spark cluster which I setup in this docker-compose file. However, I am getting the error ModuleNotFoundError: No module named 'pyspark' which doesn't make sense to me, because in this Dockerfile (which I took from the docker repos documentation) I explicitly installed pyspark with pip.

Steps to reproduce the error

# Clone the repository and checkout a specific commit
kevinsuedmersen@LAPTOP-XXX:~/dev/hadoop-sandbox$ git clone https://github.com/kevinsuedmersen/hadoop-sandbox.git
kevinsuedmersen@LAPTOP-XXX:~/dev/hadoop-sandbox$ git checkout e0a061dd3a60842aa0e93893892c7e0844c2278a

# Install and start all services
kevinsuedmersen@LAPTOP-XXX:~/dev/hadoop-sandbox$ docker-compose up -d

# Entering the container running the notebooks
kevinsuedmersen@LAPTOP-XXX:~/dev/hadoop-sandbox$ docker exec -it jupyter-spark bash

# Activating the custom python environment installed in the above referenced Dockerfile
(base) jovyan@XXX:~$ conda activate python37

# Start a jupyter notebook server
(python37) jovyan@XXX:~$ jupyter notebook

# After some logging, the following output shows
To access the notebook, open this file in a browser:
file:///home/jovyan/.local/share/jupyter/runtime/nbserver-27913-open.html
Or copy and paste one of these URLs:
http://b8ef36545270:8889/?token=some_token
or http://127.0.0.1:8889/?token=some_token

Then, I click on the URL http://127.0.0.1:8889/?token=some_token to open up the jupyter GUI in my browser, execute the simple python code snippet and get the above error as explained above.

What I've tried

To check if pyspark was really installed, I basically just tried to execute the simple python code snippet in the shell of the jupyter-spark container and surprisingly, it worked. Specifically, I executed the following commands in a new shell:

# Entering into the jupyter-spark container and activating the custom python environment
kevinsuedmersen@LAPTOP-XXX:~/dev/hadoop-sandbox$ docker exec -it jupyter-spark bash
(base) jovyan@XXX:~$ conda activate python37

# Opening a python shell
(python37) jovyan@XXX:~$ python

# Copy pasting the same commands from the notebook into the shell
>>> import pyspark
>>> from pyspark.sql import SparkSession
>>> spark = SparkSession.builder.master('spark://spark-master:7077').getOrCreate()
>>> sc = spark.sparkContext
>>> rdd = sc.parallelize(range(100 + 1))
>>> rdd.sum()
5050

Furthermore, I noticed that executing the following in the notebook

! python --version

prints Python 3.8.8

So, my question is: How can I make the notebook use the custom python environment?

jupyter-notebook pyspark apache-spark docker hadoop
1 Answers

So, apparantly, the following workaround works:

  1. Change the Dockerfile of the jupyter-spark service to something as simple as:
FROM jupyter/all-spark-notebook:584f43f06586

ARG SPARK_VERSION
ARG HADOOP_VERSION
ARG SPARK_CHECKSUM
ARG OPENJDK_VERSION
ARG PYTHON_VERSION

# Install a different version of python inside the base environment
RUN conda install -y python=$PYTHON_VERSION

# Install required pip packages, e.g. pyspark
COPY requirements.txt /docker_build/requirements.txt
RUN pip install -r /docker_build/requirements.txt
  1. The service definition in the docker-compose.yml file becomes:
# Spark notebooks
  jupyter-spark:
    # To see all running servers in this container, execute 
    # `docker exec jupyter-spark jupyter notebook list`
    container_name: jupyter-spark
    build:
      context: jupyter-spark
      args: 
        - SPARK_VERSION=3.1.1
        - HADOOP_VERSION=3.2
        - SPARK_CHECKSUM=E90B31E58F6D95A42900BA4D288261D71F6C19FA39C1CB71862B792D1B5564941A320227F6AB0E09D946F16B8C1969ED2DEA2A369EC8F9D2D7099189234DE1BE
        - OPENJDK_VERSION=11
        # Make sure the python version in the driver (the notebooks) is the same as in spark-master,
        # spark-worker-1, and spark-worker-2
        - PYTHON_VERSION=3.7.10
    ports: 
      - 8888:8888
      - 8889:8889
      - 4040:4040
      - 4041:4041
    volumes:
      - ./jupyter-spark/work:/home/jovyan/work
    pid: host
    environment: 
      - TINI_SUBREAPER=true
    env_file: 
      - ./hadoop.env
    networks:
      - hadoop

The currently working state of the repository with the above changes can be here

user_67015200
Related