Scala spark: Sum all columns across all rows

user_6016064 0
Viewed 78

I can do this quite easily with

df.groupBy().sum()

But I'm not sure if the groupBy() doesn't add additional performance impacts, or is just bad style. I've seen it done with

df.agg( ("col1", "sum"), ("col2", "sum"), ("col3", "sum"))

Which skips the (I think unnecessary groupBy), but has its own uglyness. What's the correct way to do this? Is there any under-the-hood difference between using .groupBy().<aggOp>() and using .agg?

apache-spark scala
1 Answers

If you check the Physical plan for both queries spark internally calls same plan so we can use either of them!

I think using df.groupBy().sum() will be handy as we don't need to specify all column names.

Example:

val df=Seq((1,2,3),(4,5,6)).toDF("id","j","k")

scala> df.groupBy().sum().explain
== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[sum(cast(id#7 as bigint)), sum(cast(j#8 as bigint)), sum(cast(k#9 as bigint))])
+- Exchange SinglePartition
   +- *(1) HashAggregate(keys=[], functions=[partial_sum(cast(id#7 as bigint)), partial_sum(cast(j#8 as bigint)), partial_sum(cast(k#9 as bigint))])
      +- LocalTableScan [id#7, j#8, k#9]

scala> df.agg(sum("id"),sum("j"),sum("k")).explain
== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[sum(cast(id#7 as bigint)), sum(cast(j#8 as bigint)), sum(cast(k#9 as bigint))])
+- Exchange SinglePartition
   +- *(1) HashAggregate(keys=[], functions=[partial_sum(cast(id#7 as bigint)), partial_sum(cast(j#8 as bigint)), partial_sum(cast(k#9 as bigint))])
      +- LocalTableScan [id#7, j#8, k#9]
user_76326950
Related