Why is Spark breaking my stage into 3 different stages with the same description and DAG?

Viewed 170

I have a 5 worker node cluster with 1 executor each and 4 cores per executor.

I have an rdd that is spread over 20 partitions that I check with the rdd.isEmpty method. In the spark history server, I can see three different "jobs" with the same "description":

JobId                         Description                           Tasks
3                             isEmpty at myFile.scala:42            1/1
4                             isEmpty at myFile.scala:42            4/4
5                             isEmpty at myFile.scala:42            15/15

When I click into these jobs/stages, they all have the same DAG. What might be causing the isEmpty stage to get broken into 3 different stages?

Additionally, when I change the RDD from 20 to 8 partitions, the History server shows:

JobId                         Description                           Tasks
3                             isEmpty at myFile.scala:42            1/1
4                             isEmpty at myFile.scala:42            4/4
5                             isEmpty at myFile.scala:42            3/3

In both cases the sum of tasks from these 3 stages equal the total number of partitions for the rdd. Why doesn't it just put in all in one stage like:

JobId                         Description                           Tasks
3                             isEmpty at myFile.scala:42            20/20
0 Answers
Related