BigDecimal setScale is not working in Spark UDF

user_1249627 0
Viewed 144

I was trying to cast json column to map. I tried udf for converting json to map. but it is not working as expected.

val df1 = Seq(("{\"k\":10.004}")).toDF("json")
def udfJsonStrToMapDecimal = udf((jsonStr: String)=> { var jsonMap:Map[String,Any] = parse(jsonStr).values.asInstanceOf[Map[String, Any]]
     jsonMap.map{case(k,v) => (k,BigDecimal.decimal(v.asInstanceOf[Double]).setScale(6))}.toMap
})
val f = df1.withColumn("map",udfJsonStrToMapDecimal($"json"))
scala> f.printSchema
root
 |-- json: string (nullable = true)
 |-- map: map (nullable = true)
 |    |-- key: string
 |    |-- value: decimal(38,18) (valueContainsNull = true)

instead of decimal(38,6) it converting the value as decimal(38,18) Need help here

apache-spark scala
1 Answers

The default decimal precesion is DecimalType(38, 18), I am not sure what you are trying to do but

You can cast the current decimal type as 

df1.withColumn("map",udfJsonStrToMapDecimal($"json").cast("map<string, decimal(38,6)>"))

Schema: 

root
 |-- json: string (nullable = true)
 |-- map: map (nullable = true)
 |    |-- key: string
 |    |-- value: decimal(38,6) (valueContainsNull = true)

Or you can also define the schema and read as JSON directly as 

val schema = StructType(StructField("k", DecimalType(38, 6), nullable = false) :: Nil)

val df1 = Seq(("{\"k\":10.004}")).toDF("json")

val result = df1.withColumn("value", from_json($"json", schema))

Schema: 

root
 |-- json: string (nullable = true)
 |-- value: struct (nullable = true)
 |    |-- k: decimal(38,6) (nullable = true)
user_65514260
Related