I'm submitting a job to a containerised spark cluster running locally. Spark version 3.2.1. I'm using bitnami's spark container images.
The job is written in scala. I've created a 'fat-jar'.
Now when I submit the jar to the cluster (from my local, outside the container(s)) in client mode (--deploy-mode client), providing the path to the jar in my local filesystem. The job successfully completes.
However, when I switch to cluster mode (--deploy-mode cluster) and provide a link to the jar uploaded to s3, the job fails. The error I receive is:
INFO DriverRunner: Launch Command: "/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java" "-cp" "/Users/username/Downloads/spark-3.2.1/conf:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.driver.supervise=false" "-Dspark.app.name=CsvToDelta" "-Dspark.jars=linkToS3" "-Dspark.master=spark://localhost:7077" "-Dspark.submit.deployMode=cluster" "-Dspark.submit.pyFiles=" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@172.20.0.3:46501" "/opt/bitnami/spark/work/driver-20220908212120-0004/spark-scala-csv-to-delta_2.12-1.0.jar" "CsvToDelta"
spark-worker_1 | 22/09/08 21:22:39 INFO DriverRunner: Killing driver process!
spark-worker_1 | 22/09/08 21:22:39 WARN Worker: Driver driver-20220908212120-0004 failed with unrecoverable exception: java.io.IOException: Cannot run program "/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java" (in directory "/opt/bitnami/spark/work/driver-20220908212120-0004"): error=2, No such file or directory
As you can see, it's unable to find the java executable. It's because the location it's trying to find the java is incorrect.
"/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java" is where java exists on my local (outside the container). In the container, java exists in a different path: /opt/bitnami/java/bin/java
I'm unable to understand why is spark-submit telling the driver (which I presume is running on the worker container) to find java at the location which is valid for my local, not container (why is spark-submit telling the driver where to find java at all?) Is it because of some config in the way the fat-jar is being constructed? (Should not be so because same jar works in client mode)
My spark-submit command in cluster mode looks like this:
spark-submit --deploy-mode cluster --class "CsvToDelta" --master spark://localhost:7077 'linkToS3Jar'
Command for client mode:
spark-submit --deploy-mode client --class "CsvToDelta" --master spark://localhost:7077 spark-scala-csv-to-delta_2.12-1.0.jar