Python regex - identifying words with preceding symbol

user_12036080 0
Viewed 136

I am trying to split a target sentence into composite pieces for a later function using re.split() and the regex

(@?\w+)(\W+)

Ideally, this would split words and non-word characters in a generated list, preserving both as separate list items, with the exception of the "@" symbol which could precede a word. If there is an @ symbol before a word, I want to keep it as a cohesive item in the split. My example is below.

My test sentence is as follows:

this is a test of proper nouns @Ryan

So the line of code is:

re.split(r'(@?\w+)(\W+)', "this is a test of proper nouns @Ryan")

The list that I want to generate would include "@Ryan" as a single item but, instead, it looks like this

['', 'this', ' ', '', 'is', ' ', '', 'a', ' ', '', 'test', ' ', '', 'of', ' ', '', 'proper', ' ', '', 'nouns', ' @', 'Ryan']

Since the first container has the @ symbol, I would have thought that it would be evaluated first but that is apparently not the case. I have tried using lookaheads or removing @ from the \W+ container to no avail.

https://regex101.com/r/LeezvP/1

python-3.x regex
3 Answers

With your shown samples, could you please try following(written and tested in Python 3.8.5). considering that you need to remove empty/null items in your list. This will give output where @ is together with words.

##First split the text/line here and save it to list named li.
li=re.split(r'(@?\w+)(?:\s+)', "this is a test of proper nouns @Ryan")
li
['', 'this', '', 'is', '', 'a', '', 'test', '', 'of', '', 'proper', '', 'nouns', '@Ryan']

##Use filter to remove nulls in list li.
list(filter(None, li))
['this', 'is', 'a', 'test', 'of', 'proper', 'nouns', '@Ryan']

Simple explanation would be, use split function with making 1 capturing group which has an optional @ followed by words and 1 non-capturing group which has spaces one or more occurrences in it. This will place null elements in list, so to remove them use filter function.

NOTE: As per OP's comments nulls/spaces may be required, so in that case one could refer following code; which worked for OP:

li=re.split(r'(@?\w+)(\s+|\W+)', "this is a test of proper nouns @Ryan")
user_58665800

You could also match using re.findall and use an alternation | matching the desired parts.

(?:[^@\w\s]+|@(?!\w))+|\s+|@?\w+

Explanation

  • (?: Non capture group
    • [^@\w\s]+ Match 1+ times any char except @ word char or whitespace char
    • | Or
    • @(?!\w) Match @ when not directly followed by a word char
  • )+ Close the group and match 1+ times
  • | Or
  • \s+ Match 1+ whitespace chars to keep them as a separate match in the result
  • | Or
  • @?\w+ Match @ directly followed by 1+ word chars

Regex demo

Example

import re

pattern = r"(?:[^@\w\s]+|@(?!\w))+|\s+|@?\w+"

print(re.findall(pattern, "this is a test of proper nouns @Ryan"))

# Output
# ['this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'of', ' ', 'proper', ' ', 'nouns', ' ', '@Ryan']

print(re.findall(pattern, "this @Ryan #$@test@123@4343@@$%$test@1#$#$@@@1@@@@"))

# Output
# ['this', ' ', '@Ryan', ' ', '#$', '@test', '@123', '@4343', '@@$%$', 'test', '@1', '#$#$@@', '@1', '@@@@']
user_54249880

The regex, @?\w+|\b(?!$) should meet your requirement.

Explanation at regex101:

enter image description here

1st Alternative @\w
    @ matches the character @ literally (case sensitive)
    ? matches the previous token between zero and one times, as many times as possible, giving back as needed (greedy)
    \w matches any word character (equivalent to [a-zA-Z0-9_])
    + matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)
2nd Alternative \b(?!$)
    \b assert position at a word boundary: (^\w|\w$|\W\w|\w\W)
    Negative Lookahead (?!$)
        Assert that the Regex below does not match
        $ asserts position at the end of a line
user_108195730
Related