I am running my jobs on Spark 2.2 on a cluster with Mesos via spark-submit. I declare:
- (In sparkSession config) spark.executor.cores 5
- (In spark submit config) total-executor-cores 20
- (In spark submit config) driver-cores 20
So I have 20 cores on a driver and 4 executors with 5 cores each.
Normally when I look at Stage progress bar for example
[Stage 12:================================> (86 + 20) / 100]
which indicates number of task done (86), number of tasks running (20) and total number of tasks (100). Also most of the time the number of tasks running does not exceed value given in total-executor-cores so in this case 20.
Right now however I have a job where I see something like this
[Stage 22:================================> (355 + 155) / 568]
For the same settings as above, the same cluster. In SparkUI I see 5 cores per executor (as requested) running about 30 tasks per executor at once where normally would be running at most 5 tasks (1 per each core). I don't understand why.
Could this be related to the nature of my job? It involves casting of a Spark DataFrame into a RDD and collection on driver. Example Stage
val dfOnDriver = df.select("colA", "colB", "colC")
.rdd
.map(...)
.collect
- Question 1 Why is there so many tasks running?
- Question 2 Is there a way to force this kind of behaviour on purpose in normal executor based calculations?
- Question 3 Is this kind of behaviour desired and more efficient than 1 task per core?
Let me know if you need more info about my setup.