If I have a Spark job (2.2.0) compiled with setMaster("local") what will happen if I send that job with spark-submit --master yarn --deploy-mode cluster ?
I tried this and it looked like the job did get packaged up and executed on the YARN cluster rather than locally.
What I'm not clear on:
why does this work? According to the docs, things that you set in
SparkConfexplicitly have precedence over things passed in from the command line or viaspark-submit(see: https://spark.apache.org/docs/latest/configuration.html). Is this different because I'm usingSparkSession.getBuilder?is there any less obvious impact of leaving
setMaster("local")in code vs. removing it? I'm wondering if what I'm seeing is something like the job running in local mode, within the cluster, rather than properly using cluster resources.