Convert each row of pyspark DataFrame column to a Json string

user_8340881 0
Viewed 278

How to create a column with json structure based on other columns of a pyspark dataframe.

For example, I want to achieve the below in pyspark dataframe. I am able to do this on pandas dataframe as below, but how do I do the same on pyspark dataframe

df = {'Address': ['abc', 'dvf', 'bgh'], 'zip': [34567, 12345, 78905], 'state':['VA', 'TN', 'MA']}
df = pd.DataFrame(df, columns = ['Address', 'zip', 'state'])
lst = ['Address', 'zip']

df['new_col'] = df[lst].apply(lambda x: x.to_json(), axis = 1)

Expected output enter image description here

pyspark
1 Answers

Assuming your pyspark dataframe is named df, use the struct function to construct a struct, and then use the to_json function to convert it to a json string.

import pyspark.sql.functions as F
....

lst = ['Address', 'zip']
df = df.withColumn('new_col', F.to_json(F.struct(*[F.col(c) for c in lst])))
df.show(truncate=False)
user_170214290
Related