Simple PySpark script using Dockerfile not able to run

AK91 0
Viewed 36

So I have a test.py file that creates a dataframe and outputs a csv:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([[1,2],[3,4]])
df.write.csv("test", header=True)

I'm trying to get this to run via Docker but with zero luck.

I created the following Dockerfile:

FROM apache/spark-py:v3.3.0
COPY test.py test.py
# RUN python -m pip install --upgrade pip && pip3 install pyspark==3.3.0
RUN apt-get update -y &&\
    apt-get install -y python3 &&\
    pip3 install pyspark==3.3.0
CMD ["python3", "test.py"]

and run the following to run the test.py script in Docker:

docker build -t ch_image --rm . && docker run --rm --name ch_container ch_image

I'm getting the following error message:

Sending build context to Docker daemon 31.23 kB
Step 1/4 : FROM apache/spark-py:v3.3.0
 ---> d186e5bd67b6
Step 2/4 : COPY test.py test.py
 ---> Using cache
 ---> 460491fa3ac9
Step 3/4 : RUN apt-get update -y &&    apt-get install -y python3 &&    pip3 install pyspark==3.3.0
 ---> Running in 21d6432b8d7b

Reading package lists...
E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
E: Unable to lock directory /var/lib/apt/lists/
The command '/bin/sh -c apt-get update -y &&    apt-get install -y python3 &&    pip3 install pyspark==3.3.0' returned a non-zero code: 100

I've tried:

  • COPY test.py .
  • CMD ["/opt/spark/bin/pyspark", "test.py"]
  • RUN python -m pip install --upgrade pip && pip3 install pyspark==3.3.0

Any help to understand why I'm not able to run the script would be much appreciated.

pyspark dockerfile docker
1 Answers

You got "permission denied" error. Could you please add "USER root" between step 2 and step 4 in your dockerfile as shown below?

FROM apache/spark-py:v3.3.0
COPY test.py test.py
USER root
RUN apt-get update -y &&\
    apt-get install -y python3 &&\
    pip3 install pyspark==3.3.0
CMD ["python3", "test.py"]
elademir0
Related