Removing duplicates in a dataset in python based on 2 columns simulataneously

user_13887197 0
Viewed 52

I have a pandas dataframe and wish to remove rows based on whether either entries of 2 particular columns are duplicate or not, starting from the first row and proceeding downwards (keeping the first hit and removing the other duplicates).

For example starting from:

Group_1 Group_2 Score
Red Blue 90
Red Orange 85
Red Violet 82
Green Blue 79
Green Yellow 74

And having an output where all entries are unique (considering both columns simultaneously - the fourth row is removed because "Blue" in the second column is already present in the first row):

Group_1 Group_2 Score
Red Blue 90
Green Yellow 74
pandas duplicates python
1 Answers

I'm not aware of any direct method for dropping duplicates in a col1 OR col2 manner instead of AND, but you can simply use a for loop to drop duplicates on the columns you want sequentially.

for col in ['Group_1', 'Group_2']:
    df.drop_duplicates(subset=col, inplace=True)

or

for col in df.columns[:2]:
    df.drop_duplicates(subset=col, inplace=True)

It also automatically keeps the first occurrence but you can specify it in the method: df.drop_duplicates(subset=col, inplace=True, keep='first')

user_144205080
Related