I have a spark setup currently running in PROD.
In which we are calling spark submit using shell script.
We export some variables inside shell script before spark submit in yarn client mode.
Those exported variable will refer inside scala program using "System.getenv(<variable_name_exportrd>)".
Problem: Now the problem is we are switching to yarn cluster mode in spark-submit.
If we submit that job using Cluster mode. Those exported variables coming as null inside the program.
As per below blog, if I use "spark.yarn.appMasterEnv." i am able to access those exported variables. We are exporting nearly 40 variables in shell script. So buliding --conf for 40 variables is tedious task. (Variable changes dynamically)
How to pass environment variables to spark driver in cluster mode with spark-submit
Now my question is : Is there a way to specify multiple environment variables in a file and pass that in spark-submit.
This makes code change very less.
Please help. Thanks in advance.