How to convert map column to struct type with custom key-value names?

user_6837947 0
Viewed 2029

I have a UDF that converts a Map (in this case String -> String) to an Array of Struct using the Scala built-in toArray function

val toArray = udf((vs: Map[String, String]) => vs.toArray)

The field names of structs are _1 and _2.

How can I change the UDF definition such that field (key) name was "key" and value name "value" as part of the UDF definition?

[{"_1":"aKey","_2":"aValue"}]

to

[{"key":"aKey","value":"aValue"}]
apache-spark-sql apache-spark struct scala dictionary
2 Answers

Spark 3.0+

map_entries($"col_name")

This converts a map to an array of struct with struct field names key and value.

Example:

val df = Seq((Map("aKey"->"aValue", "bKey"->"bValue"))).toDF("col_name")

val df2 = df.withColumn("col_name", map_entries($"col_name"))

df2.printSchema()
// root
//  |-- col_name: array (nullable = true)
//  |    |-- element: struct (containsNull = false)
//  |    |    |-- key: string (nullable = false)
//  |    |    |-- value: string (nullable = true)

For custom field names, just cast a new column schema:

val new_schema = "array<struct<k2:string,v2:string>>"
val df2 = df.withColumn("col_name", map_entries($"col_name").cast(new_schema))

df2.printSchema()
// root
//  |-- col_name: array (nullable = true)
//  |    |-- element: struct (containsNull = true)
//  |    |    |-- k2: string (nullable = true)
//  |    |    |-- v2: string (nullable = true)
user_27535010
Related