Remove all special characters in pandas dataframe

user_14712477 0
Viewed 3453

I'm having trouble removing all special characters from my pandas dataframe. Can you help me out?

I have tried something like this:

df = df.replace(r'\W+', '', regex=True)

because I've found it in a recent post. But when I execute, the special character " ' " for example doesn't disappear.

I know in PostgresSQL there is something like [^\w] to get a specific list. Is there something similar in python to do something like

a) keep only alphabet

b) keep only numbers

c) keep alphabet and numbers

Thank you for your help!

pandas special-characters python replace regex
2 Answers

[^0-9a-zA-Z ] matches Unicode letters and digits, this will remove too much.

Use

df = df.replace(r'[^\w\s]|_', '', regex=True)

See proof

Explanation

--------------------------------------------------------------------------------
  [^\w\s]                  any character except word characters (\p{L}, \p{N}, _) 
                           and whitespace (\p{Z})
--------------------------------------------------------------------------------
 |                        OR
--------------------------------------------------------------------------------
  _                        '_'
user_113298900

Just do this:

df = df.replace(r'[^0-9a-zA-Z ]', '', regex=True).replace("'", '')
user_87083640
Related