How to check for duplicates across multiple columns?

user_16601843 0
Viewed 212

I have a df that looks like this:

ID    Lat    Long   geo
4     23     45     xyhj
5     23     12     nil
7     40     32.    kl

If I want to check duplicates across one column, I can use

df['Lat'].is_unique

This would give me False.

But is it possible to check if there are any rows where both, the Lat and Long values are being repeated? In the case of this data frame, the answer would be True because no combination of Lat and Long is duplicated.

pandas data-analysis dataframe python-3.x python
2 Answers

For checking duplicates from the entire dataset you can use, df.duplicated().sum().

You can also explicitly write the column names and get the duplicate values.

user_143715360

You want pd.DataFrame.duplicated(subset=<list_of_columns>):

import pandas as pd
  
df_original = pd.DataFrame(
    {
        "ID": [4, 5, 7],
        "Lat": [23, 23, 40],
        "Long": [45, 12, 32],
        "geo": ["xyhj", "nil", "kl"],
    }
)
df_duplicated = pd.DataFrame(
    {
        "ID": [4, 5, 7, 8],
        "Lat": [23, 23, 40, 23],
        "Long": [45, 12, 32, 12],
        "geo": ["xyhj", "nil", "kl", "something else"],
    }
)

for df in [df_original, df_duplicated]:
    print(df, "\n", df.duplicated(subset=["Lat", "Long"]).any(), "\n\n")

This prints

   ID  Lat  Long   geo
0   4   23    45  xyhj
1   5   23    12   nil
2   7   40    32    kl 
 False 


   ID  Lat  Long             geo
0   4   23    45            xyhj
1   5   23    12             nil
2   7   40    32              kl
3   8   23    12  something else 
 True
user_17178280
Related