Java Spark - how to generate structType from a json object

user_1670951 0
Viewed 1788

How do i create a struct type out of a JSON object in java? The JSON object in my case is an AVRO schema(i have truncated it below).

{\"type\":\"record\",\"name\":\"DataRecord\",\"namespace\":\"com.mycode\",\"fields\":[{\"name\":\"data\",\"type\":{\"type\":\"record\",\"name\":\"Data\",
\"fields\":[{\"name\":\"COUNT\",\"type\":[{\"type\":\"null\"},{\"type\":\"int\"}],\"default\":null},{\"name\":\"VALUE\",\"type\":[{\"type\":\"null\"},{\"type\":\"int\"}],\"default\":null}] }

I prefer not to manually create the StructType object. I have seen ways to do it Scala, but nothing similar in Java.

avro apache-spark-sql apache-spark java
1 Answers

With "org.apache.spark" %% "spark-core" % "2.4.5" , "com.databricks" %% "spark-avro" % "3.2.0"

I was able to convert Json String Schema to AVRO Schema then to Struct Type.

import org.apache.avro.Schema;
import org.apache.spark.sql.types.StructType;
import com.databricks.spark.avro.SchemaConverters;


String schemaStr ="{ \"type\" : \"record\", \"name\" : \"test_schema\", \"namespace\" : \"com.test.avro\", \"fields\" : [ { \"name\" : \"username\", \"type\" : \"string\", \"doc\"  : \"blah blah\" }, { \"name\" : \"tweet\", \"type\" : \"string\", \"doc\"  : \"test\" }, { \"name\" : \"timestamp\", \"type\" : \"long\", \"doc\"  : \"test\" } ], \"doc:\" : \"test\" }";
Schema schema = Schema.parse(schemaStr);    
StructType requiredType = (StructType) SchemaConverters.toSqlType(schema).dataType();

Note : The newer versions of Spark / Scala have included "com.databricks" %% "spark-avro" by default

user_70945200
Related