Spark: where is the UUID() function in Scala?

YFl 0
Viewed 35

SparkSQL has the uuid() SQL built-in function. However, I couldn't find it in Scala to use with the Dataset API.

Surely, we can just go with:

expr("uuid()")

And it will work properly since it will be parsed as an expression.

Do we have it in the Scala Dataset API? Where? If not, is there any important technical reason why not?

Thanks.

Bonus question: Do we have it in the python API? Where?

pyspark apache-spark apache-spark-sql scala
1 Answers

If you take a look at Spark source code for org.apache.spark.sql.functions, uuid functions is missing here, so you can't use it via calling a scala function in dataset/dataframe api. So for now you have a few options:

  • expr("uuid()") - as you mentioned above
  • val generateUUID = udf(() => java.util.UUID.randomUUID().toString) - use UDF
userMNMs0
Related