Pandas UnicodeEncodeError: 'charmap' codec can't encode character

user_9508973 0
Viewed 4330

Working with pandas and every time a "special" char like: é comes up it throws an Encoding error. this is my code:

import pandas as pd


df = pd.read_csv(r'movies analysis\movie_metadata.csv',encoding='utf-8')
print(df.loc[df['title_year'] == 2015])

and this is the error i'm getting:

 File "D:\ANACONDA\lib\encodings\cp1255.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 645: character maps to <undefined>

there are many threads on the subject but none offer a solution that worked for me.

pandas encoding unicode python
2 Answers

What i ended up doing is "normalizing" all the characters into ascii. It's not ideal but it works for me:

import unicodedata
unicodedata.normalize('NFD', Data_U_Want_To_Normalize).encode('ascii', 'ignore')

so in the case of a letter like é it gets turned into e.

user_95089730

add export LANG=zh_CN.utf8 or export LANG=us_EN to your bashrc or your shell script, it worked for me my script is:

import pandas as pd
pd.read_excel(f) 
# there are chinese in my file
user_165529890
Related