How to differentiate between null and missing mongogdb values in a spark dataframe?

Viewed 211

Pre-requisite: The schema of MongoDB documents is unknown. It has null values and missing values: For instance CCNO in the following documents (1) has value (2) has null value (3) is missing.

I am fetching them in spark using MongoSpark.load(SparkSession, ReadConfig), but it is replacing missing values with nulls in the dataframe.

Please suggest a way to differentiate between the manually inserted nulls and spark inserted nulls(for missing values).

enter image description here

0 Answers
Related