In Spark 2.0 and above, how can we calculate/derive the size of a Shuffle Write or Shuffle Read? I know Spark UI has all that info. But how can we calcualte or derive the sizes in your script (Pyspark or Scala)? Is there a way?
0 Answers
Related
In Spark 2.0 and above, how can we calculate/derive the size of a Shuffle Write or Shuffle Read? I know Spark UI has all that info. But how can we calcualte or derive the sizes in your script (Pyspark or Scala)? Is there a way?