Drop duplicated rows by multiple columns if they originally or after exchanging position are same in Pandas

user_8410477 0
Viewed 29

Given a small dataset as follows:

  feature1   feature2  pcc
0        a          b  0.6
1        b          a  0.4
2        a          c  0.7
3        a          d -0.1
4        a          d  0.3

I would like to drop duplicated if ['feature1', 'feature2'] are originally same or after exchanging their position are same.

The expected result will be:

  feature1   feature2  pcc
0        a          b  0.6
2        a          c  0.7
3        a          d -0.1

How could I acheive that in Pandas? Thanks.

pandas dataframe python-3.x python
3 Answers

Use np.sort for sorting in numpy, asign back and then remove duplicates:

cols = ['feature1', 'feature2']
df[cols] = np.sort(df[cols],axis=1)
df = df.drop_duplicates(cols)
print (df)
  feature1 feature2  pcc
0        a        b  0.6
2        a        c  0.7
3        a        d -0.1
user_29010020

Here is an alternative, taking the set of values to use as group:

(df.groupby(df.filter(like='feature')
            .apply(lambda x: tuple(set(x)), axis=1))
   .first()
   .reset_index(drop=True)
)

output:

  feature1 feature2  pcc
0        a        b  0.6
1        a        c  0.7
2        a        d -0.1
user_163434640

Do you mean by:

df[['feature', 'feature2']] = df.filter(like='feature').sort_values(axis=1)
df = df.drop_duplicates(cols)
print(df)

Output:

  feature1 feature2  pcc
0        a        b  0.6
2        a        c  0.7
3        a        d -0.1
user_87083640
Related