I understand that UDFs are a complete blackbox to Spark and no attempt will be made to optimize it. But will the usage of Column type and its functions listed in: (https://spark.apache.org/docs/2.1.0/api/scala/index.html#org.apache.spark.sql.Column)
make the function "eligible" for Catalyst Optimizer?.
For example, UDF to create a new column by adding 1 to existing column
val addOne = udf( (num: Int) => num + 1 )
df.withColumn("col2", addOne($"col1"))
The same Function, using Column type:
def addOne(col1: Column) = col1.plus(1)
df.withColumn("col2", addOne($"col1"))
or
spark.sql("select *, col1 + 1 from df")
will there be any difference in performance between them?