Spark calls the main method on the specified class, and that main method is expected to construct a SparkContext or SparkSession. Configuration not specified explicitly, like master URL, will come from properties passed in on the command line or from configuration files.
What I'm wondering about: why does Spark look for a main method instead of using an interface like SparkJob with a run method that takes SparkSession as an injected argument? This is a common convention with other forms of managed code that run inside a container; you could still have a main method to construct and inject a SparkSession in local mode for unit testing.
In other words why not this
public class MySparkJob implements SparkJob {
public void run(SparkSession spark) {
}
}
instead of this:
public class MySparkJob {
public static void main(String[] args) {
SparkSession spark = ...
}
}