How to infer StructType schema for Spark Scala at run time given a Fully Qualified Name of a case class

user_3408256 0
Viewed 193

Since a few days I was wondering if it is possible to infer a schema for Spark in Scala for a given case class, but unknown at compile time.

The only input is a string containing the FQN of the class (that could be used for example to create an instance of the case class at runtime via reflection)

I was thinking if it was possible to do something like:

package com.my.namespace

case class MyCaseClass (name: String, num: Int)

//Somewhere else in codebase

// coming from external configuration file, so unknown at compile time
val fqn = "com.my.namespace.MyCaseClass" 


val schema = Encoders.product [ getXYZ( fqn ) ].schema

Of course, any other techniques that is not using Encoders is fine (building StructType analysing an instance of the case class ? Is it even possible ?)

What is the best approach? Is it something feasible ?

apache-spark reflection scala schema configuration
1 Answers

You can use reflective toolbox

package com.my.namespace

import org.apache.spark.sql.types.StructType
import scala.reflect.runtime
import scala.tools.reflect.ToolBox

case class MyCaseClass (name: String, num: Int)

object Main extends App {

  val fqn = "com.my.namespace.MyCaseClass" 

  val runtimeMirror = runtime.currentMirror
  val toolbox = runtimeMirror.mkToolBox()

  val res = toolbox.eval(toolbox.parse(s"""
    import org.apache.spark.sql.Encoders
    Encoders.product[$fqn].schema
  """)).asInstanceOf[StructType]

  println(res) // StructType(StructField(name,StringType,true),StructField(num,IntegerType,false))
}
user_52496210
Related