This is my script:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("test-test").getOrCreate()
sc = spark.sparkContext
import pandas
print("Hello world");
I am trying to deploy my script to my cluster, and it fails with the following error message:
bash-4.2$ spark-submit --deploy-mode client --master yarn sample.py
22/04/19 18:13:58 INFO VirtualEnvFactory: Start to setup virtualenv...
Exception in thread "main" java.util.NoSuchElementException: spark.app.id
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:246)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:246)
at org.apache.spark.SparkConf.getAppId(SparkConf.scala:452)
at org.apache.spark.api.python.VirtualEnvFactory.setupVirtualEnv(VirtualEnvFactory.scala:73)
at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:53)
at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:863)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:938)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:947)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/04/19 18:13:58 INFO ShutdownHookManager: Shutdown hook called
22/04/19 18:13:58 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-3c85e2fe-9fe9-4a15-a953-0e7fc94ac140
I've tried --deploy-mode client and cluster, no difference.
I found out that regardless of the contents of my sample.py file, I see the same exception. So does this means I will have to specify spark.app.id somewhere in the command line? I have no idea where to specify spark.app.id. I tried googling around but couldn't find anything useful.