Windows cmd encoding change causes Python crash

user_85185 0
Viewed 50643

First I change Windows CMD encoding to utf-8 and run Python interpreter:

chcp 65001
python

Then I try to print a unicode sting inside it and when i do this Python crashes in a peculiar way (I just get a cmd prompt in the same window). 

>>> import sys
>>> print u'ëèæîð'.encode(sys.stdin.encoding)

Any ideas why it happens and how to make it work?

UPD: sys.stdin.encoding returns 'cp65001'

UPD2: It just came to me that the issue might be connected with the fact that utf-8 uses multi-byte character set (kcwu made a good point on that). I tried running the whole example with 'windows-1250' and got 'ëeaî?'. Windows-1250 uses single-character set so it worked for those characters it understands. However I still have no idea how to make 'utf-8' work here.

UPD3: Oh, I found out it is a known Python bug. I guess what happens is that Python copies the cmd encoding as 'cp65001 to sys.stdin.encoding and tries to apply it to all the input. Since it fails to understand 'cp65001' it crashes on any input that contains non-ascii characters.

cmd encoding unicode windows python
10 Answers

For me setting this env var before execution of python program worked:

set PYTHONIOENCODING=utf-8
user_24285810

Do you want Python to encode to UTF-8?

>>>print u'ëèæîð'.encode('utf-8')
ëèæîð

Python will not recognize cp65001 as UTF-8.

user_597530

This is because "code page" of cmd is different to "mbcs" of system. Although you changed the "code page", python (actually, windows) still think your "mbcs" doesn't change.

user_1055970

The problem has been solved and addressed in this thread:

Change the system encoding

The solution is to deselect the Unicode UTF-8 for worldwide support in Win. It will require a restart, upon which your Python should be back to normal.

Steps for Win:

  1. Go to Control Panel
  2. Select Clock and Region
  3. Click Region > Administrative
  4. In Language for non-Unicode programs click on the “Change system locale”
  5. In popped up window “Region Settings” untick “Beta: Use Unicode UTF-8...”
  6. Restart the machine as per the Win prompt

The picture to show exact location of how to solve the issue:

How to resolve the issue

user_56520550
user_30645380

A few comments: you probably misspelled encodig and .code. Here is my run of your example.

C:\>chcp 65001
Active code page: 65001

C:\>\python25\python
...
>>> import sys
>>> sys.stdin.encoding
'cp65001'
>>> s=u'\u0065\u0066'
>>> s
u'ef'
>>> s.encode(sys.stdin.encoding)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown encoding: cp65001
>>>

The conclusion - cp65001 is not a known encoding for python. Try 'UTF-16' or something similar.

gimel_64910
Related