CSV data source does not support struct

Question

I am trying to read a CSV (tilda separated) of which one of the column is a json. When i want to convert the same to a struct type I am getting the exception.

hashes_event_struct = StructType([
    StructField("calculatedHash",StringType(), True),
    StructField("canonHash",StringType(), True),
    StructField("match",BooleanType(), True)])

hashes_metadata_struct = StructType([
    StructField("calculatedHash",StringType(), True),
    StructField("canonHash",StringType(), True),
    StructField("match",BooleanType(), True)
                ])

hashes_originator_struct = StructType([
    StructField("calculatedHash",StringType(), True),
    StructField("canonHash",StringType(), True),
    StructField("match",BooleanType(), True)
                ])

hashes = StructType([
    StructField("events",StructType(hashes_event_struct), True),
    StructField("metadata",StructType(hashes_metadata_struct), True),
    StructField("originator",StructType(hashes_originator_struct), True)
])

cryptoDetail = StructType([
     StructField("OTPSource",StringType(), True),
    StructField("hashes",StructType(hashes), True),
    StructField("highWaterMark",IntegerType(), True),
    StructField("highWaterMarkOK",BooleanType(), True),
    StructField("keyIndex",IntegerType(), True),
    StructField("payloadManifestJec",StringType(), True),
    StructField("version",StringType(), True)
])
    
crypto = StructType([
    StructField("cryptoDetail",StructType(cryptoDetail), True),
    StructField("cryptoDetailType",StringType(), True)
])

ip_schema_n = StructType([
    StructField("crypto",StructType(crypto), True),
    StructField("eventDetailType",StringType(), True),
    StructField("originatorDetailError",StringType(), True),
    StructField("rawJson",StringType(), True)
])

ipdata = 'C:\Users\snds\Downloads\input-data\posts_data_crypto.csv'
df_ds_1 = spark.read.format("csv").option("header","True").schema(ip_schema_n).option("sep","~").load(ipdata)

Below is the Exception

  135                 # Hide where the exception came from that shows a non-Pythonic
    136                 # JVM exception message.
--> 137                 raise_from(converted)
    138             else:
    139                 raise

c:\Spark3\python\pyspark\sql\utils.py in raise_from(e)

AnalysisException: CSV data source does not support struct,metadata:struct,originator:struct>,highWaterMark:int,highWaterMarkOK:boolean,keyIndex:int,payloadManifestJec:string,version:string>,cryptoDetailType:string> data type.;

The input data is as below

event,eventDetailType,originatorDetailError,rawJson
{"crypto":{"cryptoDetail":{"OTPSource":"cached","hashes":{"events":{"calculatedHash":"yyyyyyyyyy","canonHash":"yyyyyyyyy","match":true},"metadata":{"calculatedHash":"iiiiiiiiiiiii","canonHash":"kkkkkkkkkkkk","match":true},"originator":{"calculatedHash":"eeeeeeeeeee","canonHash":"ttttttttt","match":true}},"highWaterMark":24,"highWaterMarkOK":true,"keyIndex":24,"payloadManifestJec":"yyyyyy","version":"0.0.1"},"cryptoDetailType":"com.pk.KKK.domain.fish.service.version.1"}}~com.pk.fishing.domain.version.1~null~null

Is there any way i can convert it to a proper struct. Thank you for your help.

CSV data source does not support struct

0 Answers