Is a groupby transformation on data that is already partitioned wide or narrow?

user_8089695 0
Viewed 316

My understanding of Narrow and Wide transformations is as follows:

  • Narrow transformation - The data within a given partition is all that is needed to apply this transformation to the said partition and hence these transformations don't require data shuffle. example: map, filter
  • Wide transformation - The data within a given partition is not all that is needed to apply this transformation to the said partition and hence these transformations require data shuffle. example: sort

Question: If I already have my dataset partitioned then apart from sort what transformation is wide? I keep reading that groupby is wide but I don't see how. If I have all the data with a given key on a given partition (Which is how it would be if the dataset is already partitioned) then I do not need data from other partitions to apply groupby. What am I missing here?

pyspark apache-spark
1 Answers

You can get a better idea of what Spark is doing by using the explain method on a DataFrame.

Using a small example:

case class T(a: String, b: Int)
val df = Seq(T("a", 1), T("b", 2), T("a", 1)).toDF
df.groupBy("a").sum().explain

Looking at the output:

== Physical Plan ==
*(2) HashAggregate(keys=[a#11], functions=[sum(cast(b#12 as bigint))], output=[a#11, sum(b)#21L])
+- Exchange hashpartitioning(a#11, 200), true, [id=#13]
   +- *(1) HashAggregate(keys=[a#11], functions=[partial_sum(cast(b#12 as bigint))], output=[a#11, sum#25L])
      +- *(1) LocalTableScan [a#11, b#12]

The HashAggregate line is summing the values of b by key. More relevant for us is the line Exchange hashpartitioning,... which tells us that Spark is going to distribute the results. Spark doesn't know that the data is already partitioned, so it plans to hash the keys and collect them on a single partition. If your data is already partitioned then the hashing step won't actually result in any data being moved.

Other common methods which result in data shuffling are joins and aggregation methods (collect_list, count, sum, etc)

user_115826820
Related