Python Regex: Find specific phrase in any form in text (including if followed by . or ,)

user_10193156 0
Viewed 96

I'm trying to find when a specific product name is mentioned in customer notes (i.e. un-standardized, messy text). The product name is "Lending QB." Within the text, the product name can appear in any of the follow ways:

str1 ='Lending QB is a great product.'
str2 ='lending qb is great.'
str3 ='I don't think lendingqb is great.'
str4 ='I like Lending QB, but not always.'
str5 ='The best product is Lending qb.'

Here is the regex that mostly works:

df['lendingQB'] = df['Text'].str.findall('(?i)(?<!\S)lending\s?qb(?!\S)', re.IGNORECASE)

Using regex101.com to test, and confirming within my Python program, I can capture the product name in strings (str) 1-3, but not 4 and 5; which makes me believe the issue is with not finding the product name when it's followed by a punctuation mark.

My understanding is the \S would include commas and periods.

I tried adding |[,.] to the regex but then nothing matches:

'(?i)(?<!\S)lending\s?qb(?!\S|[,.])'

(I realize the IGNORECASE is redundant, but to test with regex101.com, I added the "(?i)")

Any suggestions?

AC

python regex
4 Answers

The pattern (?!\S) uses a negative lookahead to check what follows is not a non whitespace character.

What you could so is replace the (?!\S) with a word boundary \b to let it not be part of a larger match:

(?i)(?<!\S)lending\s?qb\b

Regex demo

Another way could be to use a positive lookahead to check for a whitespace character or ., or the end of the string using (?=[\s,.]|$)

For example:

str5 ="The best product is Lending qb."
print(re.findall(r'(?<!\S)lending\s?qb(?=[\s,.]|$)', str5, re.IGNORECASE))  # ['Lending qb']
user_54249880

This (?!\S) is a forward whitespace boundary.

It is really this (?![^\s]) a negative of a negative
with the added benefit of it matching at the EOS (end of string).

What that means is you can use the negative class form to add characters
that qualify as a boundary.
So, just put the period and comma in with the whitespace.

(?i)(?<![^\s,.])lending\s?qb(?![^\s,.])

https://regex101.com/r/BrOj2J/1

As a tutorial point, this concept encapsulates multiple assertions
and is basic engine Boolean class logic which speeds up the engine
by a ten fold factor by comparison.

Admin1

You have correctly identified one issue in the regex (punctuation immediately after QB), but there is a second edge case to consider given that the input is messy -- what if there are multiple spaces in Lending QB?.

I believe the most robust solution to your problem is:

(?i)(?<!\S)lending\s*qb\b
  • \b enforces that QB occur at the end of a word, automatically considering punctuation.
  • \s? was replaced with \s* to allow any amount of whitespace to be a match, rather than just zero-to-one whitespaces.

PS. Another point to consider is that \b terminates on all punctuation, (?=\s|[,.]) will only terminate on the given punctuation: , or . in this case. Given the wide range of possible punctuation (colon, semicolon, dash, hyphen, emdash...) I would strongly recommend \b over (?=\s|[,.]). Unless you want precise control over allowable terminating punctuation of course...

PPS. further test cases to illustrate my points

str6 ='Lending Qb: simply the best'
str7 ='I'm a fan of lending  QB'
user_51300780

Thank you "The fourth bird", "sln", and "Mark_Anderson". Your answers provided solutions and also were very educational. I went with Mark's answer since it seemed to be the most robust, which is where I'm trying to get to. Ideally, I do want to capture all cases when the product name is mentioned, no matter how messy it's typed.

I changed my code to this:

df['lendingQB'] = df['Text'].str.findall(r'(?i)(?<!\S)lending\s*qb\b', re.IGNORECASE)
user_101931560
Related