How to print UTF-8 encoded text to the console in Python < 3?

user_11666 0
Viewed 112410

I'm running a recent Linux system where all my locales are UTF-8:

LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
...
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

Now I want to write UTF-8 encoded content to the console.

Right now Python uses UTF-8 for the FS encoding but sticks to ASCII for the default encoding :-(

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> sys.getfilesystemencoding()
'UTF-8'

I thought the best (clean) way to do this was setting the PYTHONIOENCODING environment variable. But it seems that Python ignores it. At least on my system I keep getting ascii as default encoding, even after setting the envvar.

# tried this in ~/.bashrc and ~/.profile (also sourced them)
# and on the commandline before running python
export PYTHONIOENCODING=UTF-8

If I do the following at the start of a script, it works though:

>>> import sys
>>> reload(sys)  # to enable `setdefaultencoding` again
<module 'sys' (built-in)>
>>> sys.setdefaultencoding("UTF-8")
>>> sys.getdefaultencoding()
'UTF-8'

But that approach seems unclean. So, what's a good way to accomplish this?

Workaround

Instead of changing the default encoding - which is not a good idea (see mesilliac's answer) - I just wrap sys.stdout with a StreamWriter like this:

sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout)

See this gist for a small utility function, that handles it.

utf-8 ascii encoding shell python
5 Answers

If the program does not display the appropriate characters on the screen, i.e., invalid symbol, run the program with the following command line:

PYTHONIOENCODING=utf8 python3 yourprogram.py

Or the following, if your program is a globally installed module:

PYTHONIOENCODING=utf8 yourprogram

On some platforms as Cygwin (mintty.exe terminal) with Anaconda Python (or Python 3), simply run export PYTHONIOENCODING=utf8 and later run the program does not work, and you are required to always do every time PYTHONIOENCODING=utf8 yourprogram to run the program correctly.

On Linux, in case of sudo, you can try to do pass the -E argument to export the user variables to the sudo process:

export PYTHONIOENCODING=utf8
sudo -E python yourprogram.py

If you try this and it did no work, you will need to enter on a sudo shell:

sudo /bin/bash
PYTHONIOENCODING=utf8 yourprogram

Related:

  1. How to print UTF-8 encoded text to the console in Python < 3?
  2. Changing default encoding of Python?
  3. Forcing UTF-8 over cp1252 (Python3)
  4. Permanently set Python path for Anaconda within Cygwin
  5. https://superuser.com/questions/1374339/what-does-the-e-in-sudo-e-do
  6. Why bash -c 'var=5 printf "$var"' does not print 5?
  7. https://unix.stackexchange.com/questions/296838/whats-the-difference-between-eval-and-exec
user_49346400

While realizing the OP question is for Linux: when ending up here through a search engine, on Windows 10 the following fixes the issue:

set PYTHONIOENCODING=utf8
python myscript.py
user_25733750
Related