I simply want to clarify something that appear confusing to me on the use of threads in Spark.
Can someone confirm that in Spark:
Each Core can only run one thread or at least that a task slot correspond to one core and one core only and never a thread.
The case of the local mode is an exception. It uses available thread to simulate the available task slots in cluster mode.
You can use the SPARK_WORKER_CORES parameter in spark stand-alone mode for runnning more than one thread per core. This variable is set in the spark-env.sh file.
The word core is easy to get confused with the actual cpu core, but SPARK_WORKER_CORES parameter is not related to the cpu core.
(Looks like asked a long time ago, since I was having the same question, posting the answer here)
