UDF's vs Spark sql vs column expressions performance optimization

Viewed 2978

I understand that UDFs are a complete blackbox to Spark and no attempt will be made to optimize it. But will the usage of Column type and its functions listed in: (https://spark.apache.org/docs/2.1.0/api/scala/index.html#org.apache.spark.sql.Column)
make the function "eligible" for Catalyst Optimizer?.

For example, UDF to create a new column by adding 1 to existing column

val addOne = udf( (num: Int) => num + 1 )
df.withColumn("col2", addOne($"col1"))

The same Function, using Column type:

def addOne(col1: Column) = col1.plus(1)
df.withColumn("col2", addOne($"col1"))

or

spark.sql("select *, col1 + 1 from df")

will there be any difference in performance between them?

2 Answers
Related