Spark provide list of all columns in DataFrame groupBy

user_1219755 0
Viewed 4887

I need to group the DataFrame by all columns except "tag"

Right now I can do it in the following way:

unionDf.groupBy("name", "email", "phone", "country").agg(collect_set("tag").alias("tags"))

Is it possible to get all columns(except "tag") and pass them to groupBy method without a need to hardcode them as I do it now - "name", "email", "phone", "country".

I tried unionDf.groupBy(unionDf.columns) but it doesn't work

apache-spark-sql apache-spark scala
1 Answers

Here's one approach:

import org.apache.spark.sql.functions._

val df = Seq(
  ("a", "b@c.com", "123", "US", "ab1"),
  ("a", "b@c.com", "123", "US", "ab2"),
  ("d", "e@f.com", "456", "US", "de1")
).toDF("name", "email", "phone", "country", "tag")

val groupCols = df.columns.diff(Seq("tag"))

df.groupBy(groupCols.map(col): _*).agg(collect_set("tag").alias("tags")).show
// +----+-------+-----+-------+----------+
// |name|  email|phone|country|      tags|
// +----+-------+-----+-------+----------+
// |   d|e@f.com|  456|     US|     [de1]|
// |   a|b@c.com|  123|     US|[ab2, ab1]|
// +----+-------+-----+-------+----------+
user_63165080
Related