In this simplified example, I have a generic trait and three case classes that implement it. When using String or Integer as type parameters code works as expected, but when using Int it breaks.
trait Trait[T] {
def value: T
}
case class StrData(value: String) extends Trait[String]
case class IntData(value: Int) extends Trait[Int]
case class IntegerData(value: Integer) extends Trait[Integer]
import spark.implicits._
List(StrData("")).toDS // works
List(IntData(1)).toDS // breaks
List(IntegerData(1)).toDS // works
Breaks with:
Cause: org.codehaus.janino.InternalCompilerException: failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Incompatible return types
I blindly tried to use Encoders, ClassTags, and TypeTags but I was unable to fix this. As a workaround, I'm using the Integer type instead of Int.
Can someone explain to me why it breaks with the Int type, and how to fix it?