Opening a json column as a string in pyspark schema and working with it

user_6415630 0
Viewed 83

I have a big dataframe I cannot infer the schema from. I have a column that could be read as if each value is a json format, but I cannot know the full detail of it (i.e. the keys and values can vary and I do not know what it can be).

I want to read it as a string and work with it, but the format changes in a strange way in the process ; here is an example:

from pyspark.sql.types import *

data = [{"ID": 1, "Value": {"a":12, "b": "test"}},
        {"ID": 2, "Value": {"a":13, "b": "test2"}}
        ]
df = spark.createDataFrame(data)


#change my schema to open the column as string
schema = df.schema
j = schema.jsonValue()
j["fields"][1] = {"name": "Value", "type": "string", "nullable": True, "metadata": {}}
new_schema = StructType.fromJson(j)

df2 = spark.createDataFrame(data, schema=new_schema)
df2.show()

Gives me

+---+---------------+
| ID|          Value|
+---+---------------+
|  1| {a=12, b=test}|
|  2|{a=13, b=test2}|
+---+---------------+

As one can see, the format in column Value is now without quotes, and with = instead of : and I cannot work properly with it anymore. How can I turn that back into a StructType or MapType ?

pyspark apache-spark-sql apache-spark dataframe json
1 Answers

Assuming this is your input dataframe:

df2 = spark.createDataFrame([
    (1, "{a=12, b=test}"), (2, "{a=13, b=test2}")
], ["ID", "Value"])

You can use str_to_map function after removing {} from the string column like this:

from pyspark.sql import functions as F

df = df2.withColumn(
    "Value",
    F.regexp_replace("Value", "[{}]", "")
).withColumn(
    "Value",
    F.expr("str_to_map(Value, ', ', '=')")
)

df.printSchema()
#root
# |-- ID: long (nullable = true)
# |-- Value: map (nullable = true)
# |    |-- key: string
# |    |-- value: string (valueContainsNull = true)

df.show()
#+---+---------------------+
#|ID |Value                |
#+---+---------------------+
#|1  |{a -> 12, b -> test} |
#|2  |{a -> 13, b -> test2}|
#+---+---------------------+
user_13865510
Related