How to use collect_set and collect_list functions in windowed aggregation in Spark 1.6?

user_6544770 0
Viewed 53329

In Spark 1.6.0 / Scala, is there an opportunity to get collect_list("colC") or collect_set("colC").over(Window.partitionBy("colA").orderBy("colB")?

apache-spark-1.6 apache-spark-sql apache-spark scala
2 Answers

Existing answer is valid, just adding here a different style of writting window functions:

import org.apache.spark.sql.expressions.Window

val wind_user = Window.partitionBy("colA", "colA2").orderBy("colB", "colB2".desc)

df.withColumn("colD_distinct", collect_set($"colC") over wind_user)
.withColumn("colD_historical", collect_list($"colC") over wind_user).show(false)
user_17738410
Related