Spark Scala: functional difference in notation using $?

user_2205916 0
Viewed 142

Is there a functional difference between the following two expressions? The result looks the same to me but curious if there's an unknown unknown. What does the $ symbol indicate/how is it read?

df1.orderBy($"reasonCode".asc).show(10, false)
    
df1.orderBy(asc("reasonCode")).show(10, false)
apache-spark-sql apache-spark scala
1 Answers

Those two statements are equivalent and will lead to the identical result.

The $ notation is special for Scala Spark and is referring to an implicit StringToColumn method which interprets the subsequent string "reasonCode" as a Column

implicit class StringToColumn(val sc: StringContext) {
  def $(args: Any*): ColumnName = {
    new ColumnName(sc.s(args: _*))
  }
}

In Scala Spark you have many ways to select a column. I have written down a full list of syntax varieties in another answer on select specific columns from spark dataframe.

Using different notations do not have any impact on the performance as they all get translated to the same set of RDD instructions through Spark's Catalyst optimizer.

user_122089100
Related