I'm joining few dataframes to get a final dataframe called df and doing df.count() which gives me around 35M as count but after that, when I write the df to a location and read it again and take the count, I'm only getting 20M. This is not occurring at random as anytime I run it gives me the same numbers before and after writing. Any idea on what might be causing this issue or how to debug?
Pyspark count is giving different results before and after writing
Viewed 41
1 Answers
I found the problem, it's with one of the columns. a couple of years ago its datatype was different and I came to know about it when reading I passed an additional option "mergeSchema=True" and it threw an error.
I have explicitly specified the Schema and the problem was solved. All the records that were being dropped were the records for which that column has the old datatype
Related