Python Regex longer hyphen and letter before it

user_14610650 0
Viewed 113

I have a DataFrame where some values are stored as "Miami–Fort Lauderdale" and "Minneapolis–Saint Paul" with longer hyphen "–" (not short dash "-"). I am trying to remove them with regex in Windows command prompt, but it's not working properly.

  1. directly typing the hyphen as below does not work (werid enough):
XXX.replace(to_replace=r'\–', value=' ', regex=True)

XXX.replace(to_replace='–', value=' ')

and gives unchanged "Miami–Fort Lauderdale" and "Minneapolis–Saint Paul". Thus, I suppose for some reason cmd does not recognize hyphen.

  1. the general form is "lowercase letter + hyphen + uppercase letter" so I also tried
XXX.replace(to_replace=r'(?=[a-z]+)\W(?=[A-Z]+)', value=' ', regex=True)

interestingly this gives unchanged "Miami–Fort Lauderdale" and "Minneapolis–Saint Paul"

  1. however, the following will work
XXX.replace(to_replace=r'\W(?=[A-Z]+)', value=' ', regex=True)

and gives desired "Miami Fort Lauderdale" and "Minneapolis Saint Paul". But the problem is that this messes up other values like "Washington, D.C." into "Washington, D C." (apparently).

=====================================================

I eventually solved this by

XXX.replace(to_replace=r'\W(?=\w+\s)', value=' ', regex=True)

but I still wonder how Regex recognizes the letter before hyphen "–". It appears to me as if for some reason, a letter right before hyphen is not considered as a letter?

python regex
2 Answers

It is not entirely clear to me whether you are using regular Python, Pandas, or something else. One possible issue you might be having is that you never declared your Python script to be using UTF-8 characters. Consider the following working example:

# -*- coding: utf-8 -*-

import re

inp = "Miami–Fort Lauderdale"
output = re.sub(r'–', ' ', inp)
print(inp + "\n" + output)

This prints (as expected):

Miami–Fort Lauderdale
Miami Fort Lauderdale
user_18632290

I suggest using

.replace(to_replace=r'(?<=[a-z])[^\w\s](?=[A-Z])', value=' ', regex=True)

Here,

  • (?<=[a-z]) - a positive lookbehind that matches a location immediately preceded with a lowercase ASCII letter
  • [^\w\s] - a char other than a word and whitespace char
  • (?=[A-Z])- a positive lookahead that matches a location immediately followed with an uppercase ASCII letter.

See the regex demo online.

Also, note that \w matches an underscore, that is also a punctuation symbol. If you ever need to "include" it here, you will need an alternation:

(?<=[a-z])(?:[^\w\s]|_)(?=[A-Z])
user_38329700
Related