What does the metric "data size" (in Exchange step) represent in the Spark UI's SQL tab?

user_6580080 0
Viewed 126

What does the following 55.7 TB represent ?

enter image description here

If it is the size of the shuffle output files on disk, then what does represent the metric displayed as "Shuffle Write" in UI's Jobs tab for the given stage: enter image description here

For "data size", the documentation says "the size of broadcast/shuffled/collected data of the operator (BroadcastExchange, ShuffleExchange, Subquery)" which is not that clear...

Is it something like the size of uncompressed (lz4 algorithm by default) files ?

Thanks !

Note: Screenshots have been taken once the stage completed.

apache-spark-sql apache-spark
0 Answers
Related