Unable to recognise windowing function in intellij

Viewed 143

Unable to recognize avg and over function in IntelliJ. It is saying cannot resolve symbol avg and over. Can someone please say which library to import.

object DomainSpecificLanguage {
def main(args: Array[String]): Unit = {
System.setProperty("hadoop.home.dir", "C:/winutils")
val spark = SparkSession.builder().appName("DomainSpecificLanguage").config("spark.master" , "local").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
import spark.implicits._
val empSalary =Seq(
  Salary("sales", 1, 5000),
  Salary("personnel", 2, 3900),
  Salary("sales", 3, 4800),
  Salary("sales", 4, 4800),
  Salary("personnel", 5, 3500),
  Salary("develop", 7, 4200),
  Salary("develop", 8, 6000),
  Salary("develop", 9, 4500),
  Salary("develop", 10, 5200),
  Salary("develop", 11, 5200)).toDS
val byDepName = Window.partitionBy('depName)
empSalary.withColumn("avg", avg('salary) over byDepName)
}

}
2 Answers

Question : Unable to recognize avg and over function in IntelliJ. It is saying cannot resolve symbol avg and over. Can someone please say which library to import.

import org.apache.spark.sql.functions._

is something you are missing.. for function avg

import org.apache.spark.sql.expressions.Window 

is import for Window.

see this complete example which is working...

  package com.examples

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.expressions.Window

object DomainSpecificLanguage {
  def main(args: Array[String]): Unit = {
    System.setProperty("hadoop.home.dir", "C:/winutils")
    val spark = SparkSession.builder().appName("DomainSpecificLanguage").config("spark.master", "local").getOrCreate()
    spark.sparkContext.setLogLevel("ERROR")
    import spark.implicits._
    val empSalary = Seq(
      Salary("sales", 1, 5000),
      Salary("personnel", 2, 3900),
      Salary("sales", 3, 4800),
      Salary("sales", 4, 4800),
      Salary("personnel", 5, 3500),
      Salary("develop", 7, 4200),
      Salary("develop", 8, 6000),
      Salary("develop", 9, 4500),
      Salary("develop", 10, 5200),
      Salary("develop", 11, 5200)).toDS()
    val byDepName = Window.partitionBy('depName)
    import org.apache.spark.sql.functions._
    empSalary.withColumn("avg", avg('salary) over byDepName).show
  }

}

case class Salary(depName: String, deptnumber: Int, salary: Int)

Result :

+---------+----------+------+-----------------+
|  depName|deptnumber|salary|              avg|
+---------+----------+------+-----------------+
|  develop|         7|  4200|           5020.0|
|  develop|         8|  6000|           5020.0|
|  develop|         9|  4500|           5020.0|
|  develop|        10|  5200|           5020.0|
|  develop|        11|  5200|           5020.0|
|    sales|         1|  5000|4866.666666666667|
|    sales|         3|  4800|4866.666666666667|
|    sales|         4|  4800|4866.666666666667|
|personnel|         2|  3900|           3700.0|
|personnel|         5|  3500|           3700.0|
+---------+----------+------+-----------------+


Related