I am trying to write a Map type column from spark DF to Hive Orc table but its failing with errors "Not matching column type"
Hive Table:
CREATE EXTERNAL TABLE `default.test_map_col`(
test_col Map<String,String>)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
COLLECTION ITEMS TERMINATED BY '\002'
MAP KEYS TERMINATED BY '\003'
LINES TERMINATED BY '\n'
STORED AS ORC
LOCATION '/hdfs/path'
TBLPROPERTIES (
'serialization.null.format'='')
Schema used to create DF
schema = StructType(Seq(StructField("map_key_values", MapType(StringType, StringType), nullable = true)))
Map column populated in the inputDF as
("map_key_values", map(lit("testkey"), lit("testval"))
I also tried using a UDF to populate the map column in DF as
val toStruct = udf((c1: Map[String, String]) => c1.map {
case (k, v) => k + "\u0003" + v}.toSeq)
Any thoughts on how to write this?