I currently have a dataframe:
df
.groupBy($"letters")
.agg(collect_list($"numbers").as("numbers"))
.select(map($"letters",$"numbers").as("data"))
.agg(collect_list($"data").as("data"))
.select(to_json($"data").as("output"))
.show(false)
+------------------------------------------+
|output |
+------------------------------------------+
|[{"abc":["123","456"]},{"def":["123"]}] |
+------------------------------------------+
How can I get it into this format?
+------------------------------------------+
|output |
+------------------------------------------+
|{"abc":["123","456"],"def":["123"]} |
+------------------------------------------+
So that basically it is one map, with no [] brackets at the ends
In other words, it is currently
res34: org.apache.spark.sql.DataFrame = [data: array<map<string,array<string>>>]