I am trying to save DataFrame as text file in HDFS using spark-shell.
scala> finalDataFrame.rdd.saveAsTextFile(targetFile)
After executing the above code I found spark internally doing some work using stages.
[Stage 13:================================> (119 + 8) / 200]
I am trying to understand the basic details of this process. But not able to do so. Here my questions are -
- What is Stage 13?
- What is (119+8)/200?
[Stage 18:=============>(199 + 1) / 200][Stage 27:============> (173 + 3) / 200]
- Here what is the meaning of this line.
- Previously only 1 stage was working, but here I can found 2 stages are working. Hence when multiple stage are working in parallel?