Is there a way in pandas to remove duplicates from within a series?

user_11451039 0
Viewed 2112

I have a dataframe which has some duplicate tags separated by commas in the "Tags" column, is there a way to remove the duplicate strings from the series. I want the output in 400 to have just Museum, Drinking, Shopping.

I can't split on a comma & remove them because there are some tags in the series that have similar words like for example: [Museum, Art Museum, Shopping] so splitting and dropping multiple museum strings would affect the unique 'Art Museum' string.

Desired Output

pandas python-3.x python
5 Answers

You can split by comma and convert to a set(),which removes duplicates, after removing leading/trailing white space with str.strip(). Then, you can df.apply() this to your column.

df['Tags']=df['Tags'].apply(lambda x: ', '.join(set([y.strip() for y in x.split(',')])))
user_100670430

You can create a function that removes duplicates from a given string. Then apply this function to your column Tags.

def remove_dup(strng):
    '''
     Input a string and split them 
    '''
    return ', '.join(list(dict.fromkeys(strng.split(', '))))


df['Tags'] = df['Tags'].apply(lambda x: remove_dup(x))

DEMO:

import pandas as pd
my_dict = {'Tags':["Museum, Art Museum, Shopping, Museum",'Drink, Drink','Shop','Visit'],'Country':['USA','USA','USA', 'USA']}
df = pd.DataFrame(my_dict)
df['Tags'] = df['Tags'].apply(lambda x: remove_dup(x))
df

Output:

    Tags                          Country
0   Museum, Art Museum, Shopping    USA
1   Drink                           USA
2   Shop                            USA
3   Visit                           USA
user_73718930

Without some code example, I've thrown together something that would work.

import pandas as pd
test = [['Museum', 'Art Museum', 'Shopping', "Museum"]]
df = pd.DataFrame()
df[0] = test
df[0]= df.applymap(set)

Out[35]: 
                                0
0  {Museum, Shopping, Art Museum}
user_38064640

One approach that avoids apply

# in your code just s = df['Tags']
s = pd.Series(['','', 'Tour',
               'Outdoors, Beach, Sports', 
               'Museum, Drinking, Drinking, Shopping'])

(s.str.split(',\s+', expand=True)
      .stack()
      .reset_index()
      .drop_duplicates(['level_0',0])
      .groupby('level_0')[0]
      .agg(','.join)
)

Output:

level_0
0                            
1                            
2                        Tour
3       Outdoors,Beach,Sports
4    Museum,Drinking,Shopping
Name: 0, dtype: object
user_42384080

there maybe mach fancier way doing these kind of stuffs. but will do the job.

make it lower-case 

data['tags'] = data['tags'].str.lower()

split every row in tags col by comma it will return a list of string

data['tags'] = data['tags'].str.split(',')

map function str.strip to every element of list (remove trailing spaces). apply set function return set of current words and remove duplicates

data['tags'] = data['tags'].apply(lambda x: set(map(str.strip , x)))
user_37176580
Related