How to get dropped duplicates index of pandas dataframe

user_9898033 0
Viewed 63

Im' using df = df.drop_duplicates(["col1",["col2"]) on my pandas dataframe, but I need to know the dropped rows index, how can I do this?

pandas dataframe indexing python-3.x python
2 Answers

Use boolean indexing with mask by DataFrame.duplicated only for indices:

df = pd.DataFrame({'col1':[1] * 4, 'col2':[2,2,3,2]})
print (df)
   col1  col2
0     1     2
1     1     2
2     1     3
3     1     2
    
print (df.drop_duplicates(["col1","col2"]))
   col1  col2
0     1     2
2     1     3

mask = df.duplicated(["col1","col2"])
idx = df.index[mask]
print (idx)
Int64Index([1, 3], dtype='int64')

Or use Index.difference if already removed duplicates:

df1 = df.drop_duplicates(["col1","col2"])
idx = df.index.difference(df1.index)
print (idx)
Int64Index([1, 3], dtype='int64')
user_29010020

You can go for duplicated:

dups = df.duplicated(["col1", "col2"])
dups[dups].index

First line gives a boolean array that marks if a row is duplicated or not. Second line uses boolean indexing against itself to choose True entries and we get the indices of them.

user_93321870
Related