I'm using pyspark (Apache Spark v.3.2.1) on Databricks platform to make some data cleaning and transformations on dataset with around 150 million records. I'm getting some bizarre results which I cannot understand. Let's say that my dataframe df have columns : Column_1, Column_2, Column_3, Column_4, and I want to clean it in two steps.
df = spark.read.format("parquet").load(my_table_path)
df = df.dropDuplicates(("Column_1", "Column_2"))
logger.info("No. of records after first drop: {}".format(df.count()))
df = df.dropDuplicates(("Column_1", "Column_3"))
logger.info("No. of records after second drop: {}".format(df.count()))
Weirdly I'm getting results like (of course the difference is smaller, but I want to just make a point):
No. of records after first drop: 140000000
No. of records after second drop: 145000000
In my thinking number of rows after my second dropping of duplicates should always be lower or equal to result from first drop. What I miss?