I am running the following spark command
spark-submit --name Basic_class1 --class com.test2.Basic /Users/myusername/Downloads/project/spark/target/target-basic.jar
In the main file,
SparkSession sparkSession = SparkSession
.builder()
.appName("Basic")
.master("local")
.getOrCreate();
Here, the sparkSession overrides the appname that's provided in the command and sets it to "Basic" from "Basic_class1". However, there is a requirement where I need the non-overriden parameter "Basic_class1" in my dependecy file, but I don't have access to the main file where they set the appname. Is there anyway to get that parameter after the appname is overriden?
I know I can set some property in the properties file for this, but that would require a lot of changes in some other portion. So, that's why I was wondering if we can do it in the code itself.
Approaches I tried
I have looked on the parameters returned by sparkContext.conf().getAll(), but the appname is overriden there as well. I have been looking into the other methods in spark context and config as well, but not able to get any results. Also, didn't get any help from internet either.
Edit 1
For the spark submit command and sparksession as mentioned in question, if we do
System.out.println(sparkSession.sparkContext().appName());
then it prints "Basic". However, if we change the sparksession creation to
SparkSession sparkSession = SparkSession
.builder()
// .appName("Basic")
.master("local")
.getOrCreate();
then the
System.out.println(sparkSession.sparkContext().appName());
would print out "Basic_class1". So, if I don't give the appName in my code, whatever is there in --name is printed. But since appName is there in the file, I was wondering if I can still get the --name?