Remove duplicate words from each row in a column using lambda function

user_14280405 0
Viewed 296

I am trying to create a lambda function in order to remove duplicate words from each row of a column. I tried to define a variable as my column and to create a function that eliminates duplicate words from a sentence but I don't know how to apply that function to the entire column using lambda.

def unique_list(l):
    lst = []
    [lst.append(x) for x in l if x not in lst]
    return lst

a = 'shoes dress apple shoes mango apple'
a = ' '.join(unique_list(a.split()))  # 'shoes dress apple mango'

My column is 'dup_words', can you please help me understand how can I apply the above function to all the rows in my column using lambda?

pandas duplicates lambda python
2 Answers

Assuming that by rows and columns you mean a structure like this:

data = [
    ['shoes dress apple shoes mango apple','shoes dress apple shoes mango apple','shoes dress apple shoes mango apple'],  # Column 1
    ['shoes dress apple shoes mango apple','shoes dress apple shoes mango apple','shoes dress apple shoes mango apple'],  # Column 2
    ['shoes dress apple shoes mango apple','shoes dress apple shoes mango apple','shoes dress apple shoes mango apple']   # Column 3
]

You can use a list comprehension with a lambda function containing the function you defined:

dup_words = data[0]  # ['shoes dress apple shoes mango apple', 'shoes dress apple shoes mango apple', 'shoes dress apple shoes mango apple']

unique_words = [(lambda x: ' '.join(unique_list(x.split())))(row) for row in dup_words]  # ['shoes dress apple mango', 'shoes dress apple mango', 'shoes dress apple mango']

It can be cleaned up by changing your function to:

def unique_list(l):
    lst = []
    [lst.append(x) for x in l if x not in lst]
    return ' '.join(lst)

Then your lambda function becomes

unique_words = [(lambda x: unique_list(x.split()))(row) for row in dup_words]  # ['shoes dress apple mango', 'shoes dress apple mango', 'shoes dress apple mango']
user_136240090

If ordering doesn't matter, use set(). Short and Succint.

def unique_list(l):
    return list(set(l))


a = 'shoes dress apple shoes mango apple'
a = ' '.join(unique_list(a.split()))  # 'shoes dress apple mango'

And all hail the simple one liner:

a = 'shoes dress apple shoes mango apple'
a = ' '.join(list(set(a.split())))  # 'shoes dress apple mango'

Your new column can be written as something like:

df['deduped'] = df['some_column'].apply(lambda x: list(set(x)))
user_28292040
Related