I noticed a difference between the output of the agg() org.apache.spark.sql.Dataset function when it's called after a groupByKey.
Let’s consider the following dataset df :
+----+---------+-----+
|name| id |score|
+----+---------+-----+
| aaa| 100| 12|
| aaa| 200| 29|
| bbb| 200| 53|
| bbb| 300| 42|
+----+---------+-----+
If we run the following code in the two environments (Spark v2.3.0 and Spark v2.4.8):
df.groupByKey(_.id).agg(collect_list($"name").as[Array[Option[String]]])
We will get two different schemas as an output:
Spark 2.3.0 : [key: struct<value: int>, collect_list(name): array<string>]
Spark 2.4.8 : [value: int, collect_list(name): array<string>]
So if we want to access the value column which is the groupByKey key, we will proceed differently depending on which Spark version we’re using:
Spark 2.3.0 : df.groupByKey(_.id).agg(collect_list($"name").as[Array[Option[String]]]).select($"key.value")
Spark 2.4.8 : df.groupByKey(_.id).agg(collect_list($"name").as[Array[Option[String]]]).select($"value")
The groupByKey function gives the same result in the two environments
I couldn't find a mention of this change in the documentation. Does someone have an explanation for this breaking change?