How to convert null values in pyspark dataframe to None?

user_13754064 0
Viewed 428

I have a pyspark dataframe which connects & read data from oracle database table. This dataframe has null values in some of the columns. I need to save this dataframe to a collection in mongodB.

pyspark dataframe looks something like this

col1 col2 col3
10 5 null
null 6 'abc'

If I write this dataframe to mongodB it is saving only col1 and col2 from 1st record to a document and col2 and col3 to another document.

But I need all the 3 columns available in the mongo document for each dataframe row. I need to convert this null values to None.

Then when If I write to mongodB I can see null value column names with value as null and type as null in mongo document.

I am using MongoDB Spark Connector for writing dataframe to mongo.

Please suggest how to convert null values to None.

pyspark apache-spark pymongo mongodb
1 Answers

Use dataframe.fillna(valuetoreplace, subset=[list of columns to replace null values]) to replace the null values of your choice and then write the result to mongodb. If you want to limit to certain set of columns use subset in the below code.

df.fillna('None')

This command will replace your null values with None string in your dataframe but not as None object.

user_183708830
Related