How do I properly reduce a list of maps into one map?

Viewed 22

I currently have a dataframe:

df
.groupBy($"letters")
.agg(collect_list($"numbers").as("numbers"))
.select(map($"letters",$"numbers").as("data"))
.agg(collect_list($"data").as("data"))
.select(to_json($"data").as("output"))
.show(false)

+------------------------------------------+
|output                                    |
+------------------------------------------+
|[{"abc":["123","456"]},{"def":["123"]}]   |
+------------------------------------------+

How can I get it into this format?

+------------------------------------------+
|output                                    |
+------------------------------------------+
|{"abc":["123","456"],"def":["123"]}       |
+------------------------------------------+

So that basically it is one map, with no [] brackets at the ends

In other words, it is currently

res34: org.apache.spark.sql.DataFrame = [data: array<map<string,array<string>>>]
0 Answers
Related