In Spark, Map tasks save the output in the file at a local disk (Single Sorted and Merged Data File + Index / Map Task). This process is known as ShuffleWrite.
At the reduce stage, the Map output file is read using external shuffle service(if enabled), While reading data reduce task applies Tim sort.
But why Shuffle spill (memory) is much larger than the total amount of data read ?
