How to make spark udf accept a list with different data types?

user_4987739 0
Viewed 2113

My underlying function is defined like this:

def rowToSHA1(s: Seq[Any]): String = {
   //return sha1 of sequence
 } 
}

Here is the definition of my udf:

val toSha = udf[String, Seq[Any]](rowToSHA1)

df.withColumn("shavalue",(toSha(array($"id",$"name",$"description",$"accepted")))

It works when i pass only a list of string as parameter but i get an error when there is a boolean.

org.apache.spark.sql.AnalysisException: cannot resolve 'array(`id`, `name`, 
`description`, `accepted`)' due to data type mismatch: input to function 
array should all be the same type, but it's [string, string, string, 
boolean];;

I'm exploring the use of a generic function, is it a good idea?

FIX: converted my column to string before applying the function

df.withColumn("shavalue",(toSha(array($"id",$"name",$"description",$"accepted".cast("string)))
apache-spark user-defined-functions scala
1 Answers

The best solution I know for this kind of situation is just convert everything to String, When you read/create the DataFrame make sure everything is String or convert it at some point. Later you can convert if back to any other type.

user_16119520
Related