I am coding a Spark job in Scala and need to send some argument through command-line in JSON file format like the application name, master and some more variables.
./bin/spark-submit --name "My app" --master local[4] --conf spark.eventLog.enabled=false --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" myApp.jar
I need to send app name, master and all arguments in one JSON file like:
$SPARK_HOME/bin/spark-submit --properties-file property.conf
Is that possible? How? Can anyone please explain with a simple example?