Python regexp - remove punctuation but leave <unk> as is

user_4281353 0
Viewed 101

Question

Please advise a way to remove punctuation, but not those in <unk> or <UNK>.

For instance, from:

the asbestos fiber <unk> <unk| is < unusually <unk once it enters the <<unk>$% with

generate:

the asbestos fiber <unk> unk is unusually unk once it enters the unk with

Tried below but not as expected.

text = "the asbestos fiber <unk> <unk| is < unusually <unk once it enters the <<unk>$% with "

replacement = " "
pattern: str = '(?<!<unk)[%s%s]+(?!unk>)' % (re.escape(string.punctuation), r"\s")

re.sub(pattern=pattern, repl=replacement, string=text, flags=re.IGNORECASE).lower().strip()

Result: the asbestos fiber <unk> unk| is unusually unk once it enters the <unk> with

python regex
2 Answers

You might use a capture group to keep <unk>, match non word characters excluding whitespace chars, and replace the double whitespace chars (which can occur after the first substitution) with a single space.

(?i)(?<!\S)(<unk>)(?!\S)|[^\w\s]+

The pattern matches:

  • (?i) inline modifier for case insensitive
  • (?<!\S)(<unk>)(?!\S) Capture <unk> between whitespace boundaries in group 1 to keep in the replacement.
  • | Or
  • [^\w\s]+ Match 1+ occurrences of any char except a word or whitespace char to be deleted

Regex demo | Python demo

import re

text = "the asbestos fiber <unk> <unk| is < unusually <unk once it enters the <<unk>$% with "
pattern = r"(?i)(?<!\S)(<unk>)(?!\S)|[^\w\s]+"

print(re.sub(r"\s{2,}", ' ', re.sub(pattern, r"\1", text)).lower().strip())

Output

the asbestos fiber <unk> unk is  unusually unk once it enters the unk with
user_54249880

You can use the following regex to search for matches and replace them with a space:

(?:(?!<unk>)[\W_](?<!<unk>))+

See the regex demo.

Details:

  • (?: - start of a non-capturing group:
    • (?!<unk>) - the nex char should not be a starting char of a <unk> char sequence
    • [\W_] - any non-alphanumeric char
    • (?<!<unk>) - the previously matched char (with [\W_]) cannot be a starting char of an <unk> char sequence
  • )+ - one or more times.

See a Python demo:

import re
text = "the asbestos fiber <unk> <unk| is < unusually <unk once it enters the <<unk>$% with "
replacement = " "
pattern: str = r'(?:(?!<unk>)[\W_](?<!<unk>))+'
print( re.sub(pattern, replacement, text, flags=re.I) )
# => the asbestos fiber <unk> unk is unusually unk once it enters the <unk> with
user_38329700
Related