Is there a way to return the schema of a pyspark dataframe as a pythonic struct?

user_12011541 0
Viewed 59

I have a Pyspark Dataframe and df.schema.fields returns this:

[StructField(Id,StringType,true),
 StructField(Sub_l1,DoubleType,true),
StructField(Detail,ArrayType(StructType(List(StructField(Sub_l5,StringType,true))),false),true)]

Is there a way to get pyspark to return a Pythonic schema that I can store and reuse later, that looks like this:

 schema = StructType([
    StructField("Id", StringType(),True),
    StructField("Sub_l1", DoubleType(),True),
    StructField("Detail", ArrayType(StructType([StructField("Sub_l5", StringType(), True)]), False), True)
 ])

I see the latter on a lot of answers here, but I can't find a method to automatically generate a Pythonic version of the schema. Maybe I'm missing something obvious - I've looked at printSchema(), df.dtypes, iterating through the df.schema.fields and getting name, dataType, etc (but that misses the nested fields), json methods, and the schema methods in the pyspark documentation. If it comes down to it, I know we can recurse through the df.schema.fields to get the nested fields as well, but I'm wondering if there's a direct method to return the expected output.

pyspark struct python
1 Answers

You can get the schema as JSON with .ie

schema_out: Dict[str, Any] = df.schema.jsonValue()

and reuse it back later with

schema_in: StructType = StructType.fromJson(schema_out)

It contains all nested fields and can be serialized, isn't that what you're looking for ?

user_59860360
Related