Two separate images to run spark in client-mode using Kubernetes, Python with Apache-Spark 3.2.0?

user_8133769 0
Viewed 76

I deployed Apache Spark 3.2.0 using this script run from a distribution folder for Python:

./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build

I can create a container under K8s using Spark-Submit just fine. My goal is to run spark-submit configured for client mode vs. local mode and expect additional containers will be created for the executors.

Does the image I created allow for this, or do I need to create a second image (without the -p option) using the docker-image tool and configure within a different container ?

apache-spark kubernetes docker python
1 Answers

It turns out that only one image is needed if you're running PySpark. Using Client-mode, the code spawns the executors and workers for you and they run once you create a spark-submit command. Big improvement from Spark version 2.4!

user_81337690
Related