How to improve the speed of list comprehension on a pandas dataframe

user_11131258 0
Viewed 107

Is there a faster way to filter items from a set other than list comprehension, the list comprehension run time is a bit slow for large dataset.

I have already converted the list_stopwords to a set, which takes less time compared to a list.

             date      description
0        2018-07-18    payment receipt
1        2018-07-18    ogsg s.u.b.e.b june 2018 salar
2        2018-07-18    sal admin charge
3        2018-07-19    sms alert charge outstanding
4        2018-07-19    vat onverve*issuance 


list_stopwords = set(stop_words.get_stop_words('en'))

data['description'] =  data['description'].apply(lambda x: " ".join([word for word in x.split() if word not in (list_stopwords)]))
pandas python
1 Answers

Maybe with regex works faster:

Fist create your match case regex:


list_stopwords = set(stop_words.get_stop_words('en'))
re_stopwords= r"\b["
for word in list_stopwords: 
    re_stopwords+= "("+word+")"
re_stopwords+=r"]\b"

Now, apply on column:

data['description'] =  data['description'].apply(lambda x: re.sub(re_stopwords,'',x))

This gona replace all stopwords with ''(empty string).

I believe its faster becouse regex operate direct on the string, instead your code get a loop on a split.

user_98980330
Related