So I found out that a SparkSQL UDF unwraps an Option returned by a UDF implicitly.
scala> sqlContext.udf.register("canUnwrapOption", (n: java.lang.Integer) => Option(n).map(_ + 1))
res4: org.apache.spark.sql.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,List(IntegerType))
scala> sqlContext.sql("select canUnwrapOption(null)").show
+----+
| _c0|
+----+
|null|
+----+
scala> sqlContext.sql("select canUnwrapOption(1)").show
+---+
|_c0|
+---+
| 2|
+---+
This seems to really help where you have to handle null values. But it cannot wrap an Option implicitly.
scala> sqlContext.udf.register("cannotWrapWithOption", (n: Option[java.lang.Integer]) => n.map(_ + 1))
res8: org.apache.spark.sql.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,List(IntegerType))
scala> sqlContext.sql("select cannotWrapWithOption(1)").show
16/03/10 18:18:55 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.ClassCastException: java.lang.Integer cannot be cast to scala.Option
at $line28.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:26)
...
Why is that one is supported while the other is not?