python string replace only special characters keeping non-english alphabets

user_438223 0
Viewed 531

How can I remove only special characters from a string, but not foreign language characters. When I try the below code, it removes both special characters and non-english alphabets. But I want to remove only special characters (special characters that appear in regular English sentences).

import pandas as pd
from io import StringIO

data = """
id,name
1,A
1,B
1,C
1,D
2,E
2,F
2,ds
2,G
2, dsds
3,Endüstrisi`
"""
df = pd.read_csv(StringIO(data))

df['name'].str.replace('[^a-zA-Z\d\s]','',regex=True)

The above code results in

0            A
1            B
2            C
3            D
4            E
5            F
6           ds
7            G
8         dsds
9    Endstrisi
Name: name, dtype: object

But what I want is

The above code results in

0            A
1            B
2            C
3            D
4            E
5            F
6           ds
7            G
8         dsds
9    Endüstrisi
Name: name, dtype: object
python-3.8 pandas dataframe regex string
1 Answers

You can use

df['name'] = df['name'].str.replace(r'[^\w\s]|_', '', regex=True)

In Python 3, all regex shorthand character classes (like \w, \d, \s) are Unicode aware by default, as the re.U (re.UNICODE) flag is on by default. Thus, if you use \w construct in a negated character class, it matches all chars other than any Unicode letters, digits and _.

Since you do not want to match whitespaces, \s is added to the negated character class.

An underscore cannot be included into the negated character class (since it will not be matched then), you need an alternative to match _.

So, the pattern matches

  • [^\w\s] - any char but Unicode letters, digits, whitespaces and _
  • | - or
  • _ - an underscore.
user_38329700
Related