I have a StructType schema. I want to store it in a separate file in the same format and read it in my Spark program from that file

user_16406136 0
Viewed 171

I have the below schema, val schema = new StructType( Array( StructField("Age",IntegerType,true), StructField("Name",StringType,true), ) )

I want to keep it in a separate file in the same format and use it in my Spark program. I have seen that I can create a json format schema in a file for the same and use it in my program. But is there a way to use the same structtype format in a file and then read it?

Just to note, my schema file can contain multiple schemas like

val schema1=...
val schema2=...
val schema3=...
apache-spark scala schema
1 Answers

In pyspark, python provides eval() function. You can use this function to parse the expression to get StructType.

>>> s=eval('StructType([StructField("name",StringType())])')
>>> s
StructType(List(StructField(name,StringType,true)))

But I'm not sure if we can achieve the same thing in scala. You can try with some options mentioned here.

You can use simple DDL format to specify the schema instead of JSON.

val schema = DataType.fromDDL("Age int, name string")

val structTypeSchema = schema.asInstanceOf[StructType]

You can mention your schema in a file as shown below:

schema1 = c1 string, c2 int
schema2 = c3 int

You can read this file and create schema variables as shown below:

val prop = new Properties()
prop.load(new FileReader("/file/path"))

val schema1 = DataType.fromDDL(prop.getProperty("schema1"))
val schema2 = DataType.fromDDL(prop.getProperty("schema2"))

If you want you can use this library to parse configs instead using Properties class.

user_87733090
Related