Pass dataframe column name as parameter to the function using scala?

user_6432418 0
Viewed 947

I have a function that adds 2 columns:

def sum_num (num1: Int, num2: Int): Int = {
    return num1 + num2
}

I have a dataframe df with below values

+----+----+----+
|col1|col2|col3|
+----+----+----+
|1   |2   |5   |
|7   |4   |4   |
+----+----+----+

I want to add a column and pass column names to the function but the below code is not working. It gives error found Column required is Int

val newdf = df.withColumn("sum_of_cols1", sum_num($col1, $ col2))
              .withColumn("sum_of_cols2", sum_num($col1, $ col3))
databricks apache-spark scala
1 Answers

Change your code to:

import spark.implicits._

def sum_num (num1: Column, num2: Column): Column = {
  return num1 + num2
}

val newdf = df.withColumn("sum_of_cols1", sum_num($"col1", $"col2"))
  .withColumn("sum_of_cols2", sum_num($"col1", $"col3"))

You must operate over Spark SQL columns. You can do arithmetic operations with them. Take a look to the operators that can be used

user_68021560
Related