How to filter a typed Spark Dataset using pattern matching

user_1138523 0
Viewed 1135

I try to jump on the typed Dataset API but I'm stuck with filtering:

val ds: Dataset[(Int, Int)] = Seq((1,1)).toDS

ds.filter(ij => ij._1 > ij._2) // does work, but is not readable
ds.filter{case (i,j) => i<j} // does not work

Error:(36, 14) missing parameter type for expanded function The argument types of an anonymous function must be fully known. (SLS 8.5) Expected type was: ?

I don't understand why pattern matching does not work with filter, while it works fine with map:

ds.map{case (i,j) => i+j}
apache-spark-dataset apache-spark
3 Answers

Make it explicit:

ds.filter{x => x match { case (i,j) => i < j}}
user_89462280

This is a little bit more readable:

val ds: Dataset[(Int, Int)] = Seq((1,1)).toDS
ds.filter('_1 > '_2)

Note: you need to import spark.implicits._

user_83627650
Related