Flask URL route encoding problems

user_1458569 0
Viewed 1216

I stumbled upon a strange Unicode URL encoding problem in Flask.

@app.route("/it2/<string:name>")
def render_it2(name=""):
    name = _fix_encoding(name)
    return _render_file("it2", name)

When name contains Unicode characters, they are decoded as iso-8859-1 instead of utf-8.

So I had to add the _fix_encoding function:

def _fix_encoding(string):
    return string.encode('iso-8859-1').decode('utf8')

I understand that it is dirty hack, that's why I would like to ask Flask/Werkzeug gurus, what is wrong with my setup? What should I change in order to get URLs decoded in UTF-8 already, so that this ugly re-encoding would be not needed?

There are a lot of similar questions/answers on SO, but no responses that could be really helpful, e.g.:

I suppose that there should be some environment varialbe, or some configuration option that influences default Flask encoding.

I have the following LANG/LC_* on that host:

$ echo $LANG
en_US.UTF-8

$ export | grep LC
declare -x LC_ADDRESS="en_US.UTF-8"
declare -x LC_IDENTIFICATION="en_US.UTF-8"
declare -x LC_MEASUREMENT="en_US.UTF-8"
declare -x LC_MONETARY="en_US.UTF-8"
declare -x LC_NAME="en_US.UTF-8"
declare -x LC_NUMERIC="en_US.UTF-8"
declare -x LC_PAPER="en_US.UTF-8"
declare -x LC_TELEPHONE="en_US.UTF-8"
declare -x LC_TIME="en_US.UTF-8"

No iso-8859-1 whatsoever. Where does it come from?

flask werkzeug unicode python-3.x python
1 Answers

What application/code/web-page does create the URL in which "name" is embedded?

It is on the HTTP headers generated by that the place where "name" is already utf-8 encoded, but the HTTP side of your framework is being told it is latin1. You don't see that much because usually URLs are escaped with "%HH" - where "HH" are bytes in an UTF-8 sequence. If you can, change the source of your URLs to proper escape them (in Python, the call for that would be urllib.parse.quote(url)).

Check the official URL spec, from session 2.1 to 2.5 here: https://www.rfc-editor.org/rfc/rfc3986#section-2.1

Apparently there is no standard way to tell the HTTP server components (on the Python side, the WSGI layer and Flask itself) which is the text-encoding of the URL itself - and these components are assuming Latin-1 (AKA iso-8859-1) for the bytes outside the ASCII set used by your utf-8 data.

And finally - if quoting the URL at the source is not an option, then your approach is good enough. You probably would be better guarding your _fix_encoding function against mal-formed utf-8 data, otherwiser your application will fail. (Simply adding an errors="replace" parameter in your call to .decode will do)

user_1082050
Related