python codecs can't encode to cp1252...but notepad++ can?

user_10101412 0
Viewed 743

I have a very simple piece of code that's converting a csv....also do note i reference notepad++ a few times but my standard IDE is vs-code.

with codecs.open(filePath, "r", encoding = "UTF-8") as sourcefile:
    lines = sourcefile.read()

with codecs.open(filePath, 'w', encoding = 'cp1252') as targetfile:
    targetfile.write(lines)

Now the job I'm doing requires a specific file be encoded to windows-1252 and from what i understand cp1252=windows-1252. Now this conversion works fine when i do it using the UI features in notepad++, but when i try using python codecs to encode this file it fails;

UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in position 561488: character maps to <undefined>

When i saw this failure i was confused, so i double checked the output from when i manually convert the file using notepad++, and the converted file is encoded in windows-1252.....so what gives? Why can a UI feature in notepad++ able to do the job when but codecs seems not not be able to? Does notepad++ just ignore errors?

codec python
4 Answers

Looks like your input text has the character "�" (the actual placeholder "replacement character" character, not some other undefined character), which cannot be mapped to cp1252 (because it doesn't have the concept).

Depending on what you need, you can:

  • Filter it out (or replace it, or otherwise handle it) in Python before writing out lines to the output file.
  • Pass errors=... to the second codecs.open, choosing one of the other error-handling modes; the default is 'strict', you can also use 'ignore', 'replace', 'xmlcharrefreplace', 'backslashreplace' or 'namereplace'.
  • Check the input file and see why it's got the "�" character; is it corrupted?
user_6833290

Probably Python is simply more explicit in its error handling. If Notepad++ managed to represent every character correctly in CP-1252 then there is a bug in the Python codec where it should not fail where it currently does; but I'm guessing Notepad++ is silently replacing some characters with some other characters, and falsely claiming success.

Maybe try converting the result back to UTF-8 and compare the files byte by byte if the data is not easy to inspect manually.

Uncode U+FFFD is a reserved character which serves as a placeholder for a character which cannot be represented in Unicode; often, it's an indication of a conversion problem previously, when presumably this data was imperfectly input or converted at an earlier point in time.

(And yes, Windows-1252 is another name for Windows code page 1252.)

user_8741880

Why notepad++ "succeeds"

Notepad++ does not offer you to convert your file to cp1252, but to reinterpret it using this encoding. What lead to your confusion is that they are actually using the wrong term for this. This is the encoding menu in the program:

notepad++ screenshot

When "Encode with cp1252" is selected, Notepad decodes the file using cp1252 and shows you the result. If you save the character '\ufffd' to a file using utf8:

with open('f.txt', 'w', encoding='utf8') as f:
    f.write('\ufffd')`

and use "Encode with cp1252" you'd see three characters:

enter image description here

That means that Notepad++ does not read the character in utf8 and then writes it in cp1252, because then you'd see exactly one character. You could achieve similar results to Notepad++ by reading the file using cp1252:

with open('f.txt', 'r', encoding='cp1252') as f:
    print(f.read()) # Prints �

Notepad++ lets you actually convert to only five encodings, as you can see in the screenshot above.

What should you do

This character does not exist in the cp1252 encoding, which means you can't convert this file without losing information. Common solutions are to skip such characters or replace them with other similar characters that exist in your encoding (see encoding error handlers)

user_29078190

You are dealing with the "utf-8-sig" encoding -- please specify this one as the encoding argument instead of "utf-8".

There is information on it in the docs (search the page for "utf-8-sig").

To increase the reliability with which a UTF-8 encoding can be detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls "utf-8-sig") for its Notepad program: Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: 0xef, 0xbb, 0xbf) is written. [...]

user_2410390
Related