Spark Scala Dataset cannot use agg function

Question

I am trying to get the scala and spark work with datasets and aggregation functions.

val keyValueGroupedDataset =  dataset
  .map(mapFunctionToTheSchema)
  .groupByKey(a => a.key)

val calculatedAverage = keyValueGroupedDataset.agg(???)

Based on the mapFunctionToTheSchema (which returns multiple records such as): (key: Long, value: Long)

I was trying to calculate the average of the value for every key.

However whenever I am trying to do so

> (for instance in: 
> org.apache.spark.sql.functions.avg(myEncoder.schema("columnName").name).as("average")

Cannot resolve overloaded method 'agg'

I am not sure, what am I doing wrong, as all of the other tutorials/stackoverflows questions shows that it should work.

Ajris · Answer

I kind of answered it by myself.

I have used typed.avg[Class](yourClassElement => yourClassElement.field)

Spark Scala Dataset cannot use agg function

1 Answers