Spark - How to read and Write Struct DataSet?

Viewed 197

I have the following code:

 Dataset<Row> csvDF = spark.read().option("header", "true").csv(mycsv);
 Dataset<String> jsonDF = csvFileDF.toJSON();
 Dataset<Row> df = spark.read().schema(schema).json(spark.createDataset(jsonDF.rdd(), Encoders.STRING()));
 df.printSchema();

Schema Output as follow:

root
 |-- event_id: string (nullable = false)
 |-- user: struct (nullable = false)
 |    |-- user_id: string (nullable = true)
      |-- user_name: string (nullable = true)
 |-- store: struct (nullable = false)
 |    |-- country_code: string (nullable = false)
 |    |-- store_number: integer (nullable = false) 

When I do:

df.select("event_id").take(1)

It's working fine and i am able to see the value but for df.select("user").take(1) :

 Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4) (192.168.0.104 executor driver): java.lang.NullPointerException
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)

My goal is to write to Parquet file:

df.write().parquet("myfile");

But I am unable to do so because of struct getting same error, any help how to do this ?

0 Answers
Related