How to get count of non-duplicate elements in a column list in Pandas Dataframe?

user_1061008 0
Viewed 444

I have dug through many of SO posts to just find a Pandas solution that match my case but I could not.

The issue I have is that I have Dataframe which looks something like below:

$ df
  email               hashes  
0 user@example.com    (iz3s65inn942j1bmedv., iz3s65inn942j1bmedv., 10$0mw1ewlhqlm0l)

In my case nunique() and drop_duplicates() does not work as I need to get count of non-duplicate elements in the tuple itself. In the above case the outcome would be:

$ df
  email               hashes
0 user@example.com    1

How can I achieve this outcome and get count of non-duplicate elements of the tuple in hashes column?

pandas duplicates python-3.x
2 Answers

Use custom lambda function with Counter for count unique values only:

from collections import Counter

df['hashes'] = df['hashes'].apply(lambda x: sum(v == 1 for k, v in Counter(x).items()))
print (df)
              email  hashes
0  user@example.com       1

Pandas only alternative with DataFrame cosntructor and DataFrame.nunique:

df['hashes'] = pd.DataFrame(df['hashes'].tolist(), index=df.index).nunique(axis=1)
print (df)
              email  hashes
0  user@example.com       1
user_29010020

Define the following function, to be applied to each group of rows:

def uniqCnt(grp):
    return grp.hashes.drop_duplicates(keep=False).size

Then apply it to the "explodad" version of df, grouped by email:

result = df.explode(column='hashes').groupby('email').apply(uniqCnt).rename('hashes')

The result is the following Series:

email
user@example.com    1
Name: hashes, dtype: int64

email is the name of the index column (left column) and values are in the right column.

user_73884770
Related