Find if there are two columns with different names but identical values using pandas

user_12285101 0
Viewed 156

I have table with 30 columns, mainly numericalm with 500k rows. I would like to check if I have two columns inside this table that have the same values for all rows. for example :

I have this table:

>>> num1   num2  num3  num4
0   5.1    2.3   7     5.1
1   2.2    4.4   3.1   2.2
2   3.7    11.1   5.9  3.7
3   4.2    1.5    0.3  4.2

so in this case I would like to drop column "num4" because is identical to column "num1".

Until now I saw only ways to see if there is the same values or if they hsave the same name but not if the two columns are identical.

My end goal: to get rid of duplicated columns (by values and not by name)

pandas duplicates python
2 Answers

Try duplicated

out = df.loc[:,~df.T.duplicated()]
Out[397]: 
   num1  num2  num3
0   5.1   2.3   7.0
1   2.2   4.4   3.1
2   3.7  11.1   5.9
3   4.2   1.5   0.3

Or

out = df.T.drop_duplicates().T
Out[399]: 
   num1  num2  num3
0   5.1   2.3   7.0
1   2.2   4.4   3.1
2   3.7  11.1   5.9
3   4.2   1.5   0.3
user_79645270

The following works quite well.

import pandas as pd
data = {'ID':['1234564567', '5678995545'],'Amount': [59.99, 19.99],
        'ID2':['1234564567', '5678995545']}
data = pd.DataFrame(data)
data.T.drop_duplicates().T
           ID Amount
0  1234564567  59.99
1  5678995545  19.99

Basically you need to transpose first, drop_duplicates and transpose again.

Cheers

user_52242360
Related