Special text to latin characters in python

Question

I have the following pandas data frame:

the_df = pd.DataFrame({'id':[1,2],'name':['Joe','']})
the_df
    id  name
0   1   Joe
1   2

As you can see, we can read the second name as "Sarah", but it's written with special characters.

I want to create a new column with these characters converted to latin characters. I have tried this approach:

the_df['latin_name'] = the_df['name'].str.extract(r'(^[a-zA-Z\s]*)')
the_df
    id  name    latin_name
0   1   Joe     Joe
1   2

But it doesn't recognize the letters. Please, any help on this will be greatly appreciated.

user_6361531 · Answer

Try .str.normalize

the_df['name'].str.normalize('NFKC').str.extract(r'(^[a-zA-Z\s]*)')

Output:

       0
0    Joe
1  Sarah

user_9136348 · Answer

You can use unicodedata.normalize:

>>> import unicodedata
>>> df['name'].apply(lambda x: unicodedata.normalize('NFKD', x))
0      Joe
1    Sarah
Name: name, dtype: object