Something fun going on with unicode and python requests

Question

Firstly note that u'\xc3\xa8' is the python2 unicode string with 2 code-points, Ã and ¨. Next note that '\xc3\xa8' is the python2 byte str which represents the utf8 encoding of the character è. So u'\xc3\xa8' and '\xc3\xa8', despite looking very similar are 2 very different beasts.

Now, if we try accessing https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl in a browser all should go well.

If I define in a ipython session:

unicode_url = u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl'

then I can print it and see the same thing I entered into the URL bar of the browser, great. Let's try and fetch this with python requests.

First, I naively just try throwing the unicode url to see if requests can just deal with it: requests.get(unicode_url). Nope, 404, ok no problem, URLs should be encoded, so I try requests.get(unicode_url.encode('utf8')). Nope 404 again. No problem, maybe I need to do the URL encoding too, so I try requests.get(urllib.quote(unicode_url.encode('utf8')))....It doesn't like that at all.

However, recalling the similarities between the unicode and byte str objects I mentioned at the start, I also tried:

  requests.get('http://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premiÃ¨r-cru-brocard-75cl')

and to my surprise it works and gives a successful 200.

What is going on here with requests?

EDIT: just as another experiment (in a Scrapy shell this time)

   from scrapy.http import Request
   unicode_url = u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl'
   fetch(Request(unicode_url))

Absolutely no problems! So Why do Scrapy and the browser deal with it without issue but not python-requests? and why does the alternate url work in python-requests but not the browser or Scrapy.

Latin1 vs UTF8

It's also true that

print unicode_url.encode('utf8').decode('latin1')
u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premiÃ¨r-cru-brocard-75cl'

In general, I belive it's true that for latin unicode chars only, if you have a unicode str like u'\xe8' then you can convert it to a byte str of the same form by encoding to latin1, i.e. u'è'=u'\xe8' and u'\xe8'.encode('latin1') = '\xe8' (the object on the right being a byte str encoding in latin1 with the same form as the unicode code point representing the è)

So

In [95]: print u'è'.encode('utf8').decode('latin1')
Ã¨

and similarly,

In [94]: print u'Ã¨'.encode('latin1').decode('utf8')
è

I wonder if the culprit is

def prepare_url(self, url, params):
    """Prepares the given HTTP URL."""
    #: Accept objects that have string representations.
    #: We're unable to blindly call unicode/str functions
    #: as this will include the bytestring indicator (b'')
    #: on python 3.x.
    #: https://github.com/kennethreitz/requests/pull/2238
    if isinstance(url, bytes):
        url = url.decode('utf8')
    else:
        url = unicode(url) if is_py2 else str(url)

from requests/models.py.

Something fun going on with unicode and python requests

2 Answers