Duplicated() function as boolean indexing generates different result compares with drop_duplicates

user_10034582 0
Viewed 113

The data frame looks like the following:

df = pd.DataFrame({'k1':['one']*3 + ['two']*4,'k2':[1,1,2,3,3,4,4]})

When I am checking duplicates, I get boolean index by doing df.duplicated(), then I use it as the filter df[df.duplicated()] which shows different result compares with df.drop_duplicates()

An additional row has been created in the result

2 one 2
pandas transform python
1 Answers

drop_duplicate will drop all duplicated row . duplicated will return False for the first item and True for the other row of duplicates when it has the duplicate , so they are different function target for different problem .

df.duplicated()
0    False
1     True
2    False
3    False
4     True
5    False
6     True
dtype: bool

df.drop_duplicates()
    k1  k2
0  one   1
2  one   2
3  two   3
5  two   4

How to make the output same ?

Check the unique value 

df[~df.duplicated(keep=False)]
    k1  k2
2  one   2

df.drop_duplicates(keep=False)
    k1  k2
2  one   2
user_79645270
Related