Any way to get non-overriden spark config from command

Viewed 36

I am running the following spark command

spark-submit --name Basic_class1 --class com.test2.Basic /Users/myusername/Downloads/project/spark/target/target-basic.jar

In the main file,

SparkSession sparkSession = SparkSession
                .builder()
                .appName("Basic")
                .master("local")
                .getOrCreate();

Here, the sparkSession overrides the appname that's provided in the command and sets it to "Basic" from "Basic_class1". However, there is a requirement where I need the non-overriden parameter "Basic_class1" in my dependecy file, but I don't have access to the main file where they set the appname. Is there anyway to get that parameter after the appname is overriden?

I know I can set some property in the properties file for this, but that would require a lot of changes in some other portion. So, that's why I was wondering if we can do it in the code itself.

Approaches I tried

I have looked on the parameters returned by sparkContext.conf().getAll(), but the appname is overriden there as well. I have been looking into the other methods in spark context and config as well, but not able to get any results. Also, didn't get any help from internet either.

Edit 1

For the spark submit command and sparksession as mentioned in question, if we do

System.out.println(sparkSession.sparkContext().appName());

then it prints "Basic". However, if we change the sparksession creation to

SparkSession sparkSession = SparkSession
                .builder()
//                .appName("Basic")
                .master("local")
                .getOrCreate();

then the

System.out.println(sparkSession.sparkContext().appName());

would print out "Basic_class1". So, if I don't give the appName in my code, whatever is there in --name is printed. But since appName is there in the file, I was wondering if I can still get the --name?

0 Answers
Related