Difficult JSON To Convert To A Spark Dataframe

user_4235840 0
Viewed 50

I've been trying to load a JSON into a pyspark dataframe but I'm struggling a bit here.

This is what I have tried so far (with and without multiline):

import json
newJson = json.dumps(testjson)
newdf = spark.read.json(sc.parallelize([newJson]))
newdf.display()

JSON File:

testjson = [
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
]

When trying to display the dataframe, I get "corrupt_record". What am I doing wrong?

pyspark apache-spark-sql apache-spark json
1 Answers

Try converting it to a list of strings. Spark cannot understand a list of tuple of strings. Also json.dumps is unnecessary because Spark should be able to understand your json input.

df = spark.read.json(sc.parallelize([i[0] for i in testjson]))

df.show(truncate=False)
+--------------------------------------------------------------------+---+
|address                                                             |id |
+--------------------------------------------------------------------+---+
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
+--------------------------------------------------------------------+---+
user_141657300
Related