Why does Spark call "main" instead of using an interface like "SparkJob"?

Viewed 118

Spark calls the main method on the specified class, and that main method is expected to construct a SparkContext or SparkSession. Configuration not specified explicitly, like master URL, will come from properties passed in on the command line or from configuration files.

What I'm wondering about: why does Spark look for a main method instead of using an interface like SparkJob with a run method that takes SparkSession as an injected argument? This is a common convention with other forms of managed code that run inside a container; you could still have a main method to construct and inject a SparkSession in local mode for unit testing.

In other words why not this

public class MySparkJob implements SparkJob {
  public void run(SparkSession spark) {
  }
}

instead of this:

public class MySparkJob {
  public static void main(String[] args) {
     SparkSession spark = ...
  }
}
0 Answers
Related