I have the following code:
Dataset<Row> csvDF = spark.read().option("header", "true").csv(mycsv);
Dataset<String> jsonDF = csvFileDF.toJSON();
Dataset<Row> df = spark.read().schema(schema).json(spark.createDataset(jsonDF.rdd(), Encoders.STRING()));
df.printSchema();
Schema Output as follow:
root
|-- event_id: string (nullable = false)
|-- user: struct (nullable = false)
| |-- user_id: string (nullable = true)
|-- user_name: string (nullable = true)
|-- store: struct (nullable = false)
| |-- country_code: string (nullable = false)
| |-- store_number: integer (nullable = false)
When I do:
df.select("event_id").take(1)
It's working fine and i am able to see the value but for df.select("user").take(1) :
Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4) (192.168.0.104 executor driver): java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
My goal is to write to Parquet file:
df.write().parquet("myfile");
But I am unable to do so because of struct getting same error, any help how to do this ?