Converting dictionary to a Data Frame in Pyspark

user_17500571 0
Viewed 285

Description . How can I convert a dictionary dataset to DataFrame in PySpark:

Error/Not expected result I tried

df = spark.createDataFrame([Row(**i) for i in bounds])

but get:

TypeError: Can not infer schema for type: <class 'str'>

this code :

rdd = sc.parallelize(bounds)
rdd.map(lambda x: (x,)).toDF().show()

and others give unexpected result.

Expected result:

enter image description here

My DataSet:

enter image description here

pyspark apache-spark-sql
1 Answers

Your input to createDataFrame() has incorrect format. It should look like this -

[("price", {"q1":1, "q3": 3, "upper": 10, "lower":2} ), 
 ("carAge", {"q1":1, "q3": 3, "upper": 11, "lower":1})]

This is a list of tuples (a list of lists would also work), where each tuple has two elements, the first is string, and the second is dictionary. Each tuple contains all row data in the future spark dataframe, and two elements in tuple means there will be 2 columns in the dataframe you'll create.

To bring your dictionary data to the format above, use this line of code:

[(x, dct[x]) for x in dct.keys()]

where dct is your original dictionary as in My DataSet image. Then, you can create spark dataframe as follows:

df = (spark.createDataFrame([(x, dct[x]) for x in dct.keys()], 
                                                   schema=["Colums", "dct_col"]))

This dataframe will have only two columns, the second column, "dct_col" will be the dictionary column, and you can get "q1" , "q3", and other columns as follows:

df_expected_result = (df
                      .withColumn("q1", df.dct_col["q1"])
                      .withColumn("q3", df.dct_col["q3"])
                      .withColumn("lower", df.dct_col["lower"])
                      .withColumn("upper", df.dct_col["upper"]))
user_100194640
Related