pyspark: drop duplicates with exclusive subset

user_7009990 0
Viewed 185

I can use df1.dropDuplicates(subset=["col1","col2"]) to drop all rows that are duplicates in terms of the columns defined in the subset list.

Is it possible to have the same result by specifying the columns not to include in the subset list (something like df1.dropDuplicates(subset=~["col3","col4"])?

Thanks

pyspark apache-spark-sql apache-spark
1 Answers
df1.dropDuplicates(subset=[col for col in df1.columns if col not in ["col3","col4"]])
user_141657300
Related