I'm trying to run Apache Spark in local mode by calling spark-submit. This is done for profiling purposes. The following three parameters confuse me:
--master local[K]
--conf spark.executor.cores=X
--conf spark.executor.instances=Y
My understanding is that given sufficient amount of physical cores to use and allowing Spark to use as many threads as possible (by local[*]), there will be at least X*Y number of threads running the executors. But how does --master local[K] affect this? Will the number of threads be capped by a smaller K? On the utilization of what resource does K put a limit? I hope people who have experience in Spark can offer me some clarification.
Thanks.