I am trying to setup and use a local Spark development environment using docker-compose. I created a docker-compose stack made up of 3 services:
- a Spark master using bitnami/spark
- one Spark worker using bitnami/spark
- a development container with Scala and sbt (based on hseeberger/scala-sbt)
Working in the development container, I created a very simple HelloWorld scala app and built it using sbt assembly, setting the spark-core dependency with Provided scope.
Now I want to run the HelloWorld jar as a job on my small Spark cluster. I managed to run the job with --deploy-mode cluster by copying the jar on the Spark worker container and executing spark-submit from the worker itself. The job logs appear correctly on the Spark browser UI. But it is this the right way?
How would I do it with more workers? I tried to spark-submit the job from the Spark master but it only works in --deploy-mode=client and in such case the job logs don't appear on the browser UI. If I try running it with --deploy-mode cluster it gives me a File Not Found error on the HelloWorld jar.
To sum up
- I am getting confused with the difference of client/cluster deploy mode
- I am not sure about where (i.e. which node) the
spark-submitcommand should be executed - I am not understanding if I am supposed to copy the jar manually on each worker or if there's a better way to distributed the jar file
Thank you for your help!