Python - How to delete all characters in a sub string up to and including a keyword

user_11489646 0
Viewed 69

I have a fairly large corpus (500k new articles) in a dataframe in one column. The beginning of most (tho not all) articles has various random throw away text up to the phrase '(Reuters) - '

I have tried various permutations of the following regex trying to adjust the entire column in one shot with no luck as it either deletes chunks of the articles or does nothing.

r = re.compile(r'\A\b.*[Reuters]\b')
reuters3 = reuters2['story_text'].str.replace(r,'', regex=True)

Any ideas on how best to tackle this from a regex and pandas method perspective? thank you

Provided below is an example showing the general pattern of the text to be removed at the beginning (up to and including (Reuters) -), to keep in the middle get rid of and to get ride of at the end (everything following and including (Editing by...). The exact language, characters and length varies considerably tho across articles outside of these key cut off words.

By Chris Scicluna VALLETTA, Jan 1 (Reuters) - The Mediterranean island of Malta became the smallest member of the euro zone at the stroke of midnight on Tuesday....[various lines of article text]...public information campaign has been a widely acknowledged success. (Editing by Michael Winfrey) ((gavin.jones@reuters.com; +39-06-8522-4232; Reuters Messaging: gavin.jones.reuters.com@reuters.net)) Keywords: ECB EXPANSION/EURO MALTA

pandas text python regex
2 Answers

If you need to keep the word, you can use

reuters2['story_text'].str.replace(r'(?s)^.*?(?=\(Reuters\)\s*-)', '')

If you don't need to keep the word, you can use

reuters2['story_text'].str.replace(r'(?s)^.*?\(Reuters\)\s*-\s*', '')

Or, use Series.str.split like this:

import pandas as pd
df = pd.DataFrame({'story_text':['Some rubbish ... (Reuters) - Text']})
df['story_text'].str.split(r'\(Reuters\)\s*-', n=1).str[-1]
# => 0     Text

Details

  • (?s) - DOTALL modifier that makes . match any char
  • ^ - start of a string
  • .*? - any 0 or more chars as few as possible
  • \(Reuters\) - a literal (Reuters) text
  • (?=\(Reuters\)\s*-) - a positive lookahead that matches a location immediately followed with (Reuters), 0+ whitespaces and -
  • \s*-\s* - - enclosed with 0+ whitespaces.

See the regex demo #1 and regex demo #2.

The split solution is using a much simpler regex, \(Reuters\)\s*- and splits the string into 2 parts (since the n=1 is defined, n is the number of splits) and .str[-1] gets the last (second here) item.

user_38329700

Just .split() on it

parts = starting_string.split("Reuters", 1)  # split at most once
story = parts[-1]  # get the last part

example

>>> s = "blah blah Reuters bulk of the story"
>>> s.split("Reuters", 1)
['blah blah ', ' bulk of the story']
>>> "missing the newsgroup!".split("Reuters", 1)
['missing the newsgroup!']
>>> ["start", "end"][-1]
'end'
>>> ["bulk without splitword"][-1]
'bulk without splitword'

adding a space or other chars around your split target may help too

all together:

>>> s = "blah blah Reuters bulk of the story"
>>> s.split(" Reuters ", 1)[-1]
'bulk of the story'

You may want to do some extra validation against the possible case that your splitting string isn't simply mentioned somewhere in an article which does not have it in the header. Perhaps simply that if there were two parts, the second is longer than the first and at most N characters.

user_45410450
Related