Spark SQL : is it possible to read the custom schema from an external source instead of creating it in within the spark code?

user_10644568 0
Viewed 809

Trying to load a csv file without schema inference. Usually we create the schema as StructType within the spark code. Is it possible to save the schema in an external file (may be a property/config file) and read it dynamically while creating the dataframe ?

val customSchema_v2 = new StructType()
  .add("PROPERTY_ID_2222", "int" )
  .add("OWNER_ID_2222", "int")

Is it possible to save the schema i.e "PROPERTY_ID_2222", "int" and "OWNER_ID_2222", "int" in a file and call the schema from there ?

apache-spark-sql apache-spark scala
2 Answers

Both StructType and StructField can Serializable, so you can serialize a StructType to a file and deserialize it when you need

user_106438390

You can use JSON for schemas.

import org.apache.spark.sql.types._

val customSchema_v2 = new StructType()
                         .add("PROPERTY_ID_2222", "int" )
                         .add("OWNER_ID_2222", "int")

val schemaString = customSchema_v2.json

println(schemaString)

val loadedSchema = DataType.fromJson(schemaString)

CONSOLE Output:

{"type":"struct","fields":[{"name":"PROPERTY_ID_2222","type":"integer","nullable":true,"metadata":{}},{"name":"OWNER_ID_2222","type":"integer","nullable":true,"metadata":{}}]}

You would need to add code that reads the schema from the JSNO files.

JSON files could also be created manually and can be in pretty format. To understand it better add more columns with different data types and use customSchema_v2.prettyJson to lear the syntax.

user_14858800
Related