How to save struct column as string to CSV/TSV in PySpark?

Viewed 24

I've seen similar questions asked many times, but there's no clear answer to something that should be easy.

How can a struct column be saved to CSV (tsv actually) in PySpark? I want to serialize it and save as JSON.

I have a dataframe, which I read from parquet, that contains the following schema:

timestamp:long
timezoneOffset:string
dayInterval:integer
speed:double
heading:double
ignitionStatus:integer
segmentId:string
pointMM:struct
    mmResult:array
        element:struct
            primitiveId:long
            rnId:integer
            internalId:integer
            isFromTo:boolean
            offset:double
            probability:double
            distanceToArc:double
            headingDifference:double
    isSuccessful:boolean

The pointMM column is a struct, that contains an array of structs, and another bool field (isSuccessful). I'm able to read this data from parquet and preview it: enter image description here

If I want to save this data to CSV/TSV I get the following error:

df.write.csv(output_path, sep='\t')

AnalysisException: CSV data source does not support struct<mmResult:array<struct<primitiveId:bigint,rnId:int,internalId:int,isFromTo:boolean,offset:double,probability:double,distanceToArc:double,headingDifference:double>>,isSuccessful:boolean> data type.

Is there a way, even better an easy to convert the pointMM column to JSON string and save it to TSV?

Is there a way to do it with explicitly stating the schema of pointMM? Or a way to do it, even better, without knowing the schema?

I don't understand why this is difficult, because as you can see in the attached screenshot, that column is shown in JSON format.

EDIT 1: I understand that display() function somehow serializes the columns struct. Is there a way to use the same serialization without reinventing the wheel?

EDIT 2: .printSchema() shows the schema of the DataFrame. Can this be somehow used to help serialize pointMM column to JSON?

printing schema

print schema output

0 Answers
Related