Details of Stage in Spark

Viewed 1853

I am trying to save DataFrame as text file in HDFS using spark-shell.

scala> finalDataFrame.rdd.saveAsTextFile(targetFile)

After executing the above code I found spark internally doing some work using stages.

[Stage 13:================================> (119 + 8) / 200]

I am trying to understand the basic details of this process. But not able to do so. Here my questions are -

  1. What is Stage 13?
  2. What is (119+8)/200?

[Stage 18:=============>(199 + 1) / 200][Stage 27:============> (173 + 3) / 200]

  1. Here what is the meaning of this line.
  2. Previously only 1 stage was working, but here I can found 2 stages are working. Hence when multiple stage are working in parallel?
1 Answers
Related