Most efficient way to remove list elements from column

user_11037629 0
Viewed 78

I have a large DataFrame df with 40k+ rows:

                   words
0  coffee in the morning
1  good morning
2  hello my name is
3  hello world
...

I have a list of English stopwords that I want to remove from the column, which I do as follows:

df["noStopwords"] = df["words"].apply(lambda x: [i for i in x if i not in stopwords])

      noStopwords
0  coffee morning
1  good morning
2  hello name
3  hello world
...

This works but takes too long. Is there an efficient way of doing this?

EDIT:

print(stopwords)

['a', "a's", 'able', 'about', 'above', 'according', 'accordingly', 'across', ...]
pandas dataframe python-3.x python
1 Answers

Not sure if it's any faster, but you could try str.replace. Since words is actually a list, we also have to join and split as well.

import re
pattern = rf"\b(?:{'|'.join(map(re.escape, stopwords))})\b"
df['noStopwords'] = (df.words
    .str.join(' ')
    .str.replace(pattern, '', regex=True)
    .str.split('\s+')
)
user_131383640
Related