Create DataSet for case class that implements a generic Trait

Viewed 82

In this simplified example, I have a generic trait and three case classes that implement it. When using String or Integer as type parameters code works as expected, but when using Int it breaks.

trait Trait[T] {
  def value: T
}

case class StrData(value: String) extends Trait[String]
case class IntData(value: Int) extends Trait[Int]
case class IntegerData(value: Integer) extends Trait[Integer]
import spark.implicits._
List(StrData("")).toDS // works
List(IntData(1)).toDS  // breaks
List(IntegerData(1)).toDS  // works

Breaks with:

Cause: org.codehaus.janino.InternalCompilerException: failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Incompatible return types

I blindly tried to use Encoders, ClassTags, and TypeTags but I was unable to fix this. As a workaround, I'm using the Integer type instead of Int.

Can someone explain to me why it breaks with the Int type, and how to fix it?

0 Answers
Related