I have a map of type Map[String, String] which i want to append within another map2 of type Map[String, Any]. Later this map2 will be added as a column in a dataframe.
When I try to add map2 as a column using spark.sql.functions.typedLit, it fails with error:
Unsupported literal type class scala.collection.mutalbe.HashMap Map...
Any suggestion to achieve this?
Spark SQL API does not support Any in the schema, so you get the same error as in the snapshot below while creating the df3 . Map has to be one of the supported type by Spark SQL.
import spark.sqlContext.implicits._
import org.apache.spark.sql.functions._
val df = Seq(("foo", 1), ("bar", 2), ("baz", 3)).toDF("str", "num")
val map1 = Map("foo" -> 42, "bar" -> 3)
val df2 = df.withColumn("typedLit_map", typedLit(map1))
df2.show()
val map2: Map[String, Any] = Map("foo" -> 42, "bar" -> 3.14, "baz" -> "xyz")
val df3 = df.withColumn("typedLit_map", typedLit(map2))