How to remove all words occuring before a stop word

user_16196437 0
Viewed 106

I have a dataframe containing platform terms (platform + 3 words before):

Paper A Paper B
at a digital platform add a digital platform
change the consumer platform got a feedback platform

For each string in the dataframe I want to delete the stopwords and any word that is occuring in front of the stop word.

Dataframe should look like this:

Paper A Paper B
digital platform digital platform
consumer platform feedback platform

My best try so far:

import regex as re
def remove_stops_and_words_before(doc):
            stopwords = re.compile(r"\L<words>", words=['n‘t',  'during', 'from', 'elsewhere', 'herself', 'anyone',...]
            rx_to_last = r'^.*{}'.format(re.escape(stopwords))
            doc = re.sub(rx_to_last, '', doc, flags=re.DOTALL).strip()
            return doc
            
df_new = df.applymap(remove_stops_and_words_before)

python regex
1 Answers

You need to reconsider the way you deal with the word lists and the pattern you use.

Here is a possible solution with the regular re package:

# Sample stopwords
stopwords = ['n‘t',  'during', 'from', 'elsewhere', 'herself', 'anyone', 'at', 'a', 'an', 'the']
import re
rx = r"\b(?:\w+\W+)?(?:{})\b".format("|".join(map(re.escape, stopwords)))
# => \b(?:\w+\W+)?(?:n‘t|during|from|elsewhere|herself|anyone|at|a|an|the)\b
df.apply(lambda x: x.str.replace(rx, '', regex=True).str.strip())
# =>               Paper A             Paper B
#    0    digital platform    digital platform
#    1   consumer platform   feedback platform

See the regex demo. Details:

  • \b - word boundary (works if all your stopwords start with a word char, letter/digit/_)
  • (?:\w+\W+)? - an optional sequence of one or more word chars and then one or more non-word chars
  • (?:n‘t|during|from|elsewhere|herself|anyone|at|a|an|the) - one of the stopwords
  • \b - a word boundary (works if all your stopwords end with a word char, letter/digit/_).

If you really want to remove any text before the last occurrence of a stopword, you'd need

rx = r"(?s).*\b(?:{})\b".format("|".join(map(re.escape, stopwords)))
user_38329700
Related