I am using the spark java api and I have come to notice this weird thing that I can't explain. As you can see in
which is the dag visualization of my program execution, no other stage uses the computation of stage 3 , also the three operation in stage 3 are exactly the first 3 operations of stage 2, so my question , why is stage 3 computed separately ? I have also run the program without the last join operation , which gives the following dag,
notice here that there is no parallel stage like in the previous one . I believe that due to this unexplained stage 3 , my program is slowing down .
PS : I am very new to spark and this is my first stackoverflow question, please let me know if it's off topic or requires more detail.