I am using Spark 2.4 and trying to submit a job on a kubernetes cluster. I used this tutorial to set up the service accounts and authentication for my cluster: https://www.oak-tree.tech/blog/spark-kubernetes-primer
The code below is my spark-submit code which I am running inside my docker container:
/opt/spark/bin/spark-submit --name sparkpi-test1 \
--master k8s://<k8s-apiserver-host-for-my-cluster>:<my-port> \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--conf spark.kubernetes.driver.pod.name=$SPARK_DRIVER_NAME \
--conf spark.kubernetes.authenticate.subdmission.caCertFile=$K8S_CACERT \
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=$K8S_TOKEN \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=$SA \
--conf spark.kubernetes.namespace=$SPARK_NAMESPACE \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.container.image=$DOCKER-IMAGE \
--conf spark.kubernetes.container.image.pullPolicy=Always \
local:///opt/spark/work-dir/program_code/data/code.py
However, this generates the following error:
Exception in thread "main" java.io.FileNotFoundException: (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.spark_project.guava.io.Files$FileByteSource.openStream(Files.java:124)
at org.spark_project.guava.io.Files$FileByteSource.openStream(Files.java:114)
at org.spark_project.guava.io.ByteSource$AsCharSource.openStream(ByteSource.java:287)
at org.spark_project.guava.io.CharSource.read(CharSource.java:136)
at org.spark_project.guava.io.Files.toString(Files.java:369)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$$anonfun$5.apply(SparkKubernetesClientFactory.scala:74)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$$anonfun$5.apply(SparkKubernetesClientFactory.scala:73)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$OptionConfigurableConfigBuilder$$anonfun$withOption$extension$1.apply(SparkKubernetesClientFactory.scala:98)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$OptionConfigurableConfigBuilder$$anonfun$withOption$extension$1.apply(SparkKubernetesClientFactory.scala:97)
at scala.Option.map(Option.scala:146)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$OptionConfigurableConfigBuilder$.withOption$extension(SparkKubernetesClientFactory.scala:97)
at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:73)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
The java.io.FileNotFoundException error does not tell me in which file the problem, so I really do not understand where the problem is.
Also, no matter how I change the spark-submit code, I still get the same error.