I have a 5 worker node cluster with 1 executor each and 4 cores per executor.
I have an rdd that is spread over 20 partitions that I check with the rdd.isEmpty method. In the spark history server, I can see three different "jobs" with the same "description":
JobId Description Tasks
3 isEmpty at myFile.scala:42 1/1
4 isEmpty at myFile.scala:42 4/4
5 isEmpty at myFile.scala:42 15/15
When I click into these jobs/stages, they all have the same DAG. What might be causing the isEmpty stage to get broken into 3 different stages?
Additionally, when I change the RDD from 20 to 8 partitions, the History server shows:
JobId Description Tasks
3 isEmpty at myFile.scala:42 1/1
4 isEmpty at myFile.scala:42 4/4
5 isEmpty at myFile.scala:42 3/3
In both cases the sum of tasks from these 3 stages equal the total number of partitions for the rdd. Why doesn't it just put in all in one stage like:
JobId Description Tasks
3 isEmpty at myFile.scala:42 20/20