str.encode() giving unexpected results

user_15238797 0
Viewed 124

I've been playing around with python built-ins and have gotten some confusing (for me) results.

Take a look at this code:

>>> 'ü'.encode()
b'\xc3\xbc'

Why was \xc3\xbc (195 and 188 in decimal) returned? If you look at the ascii table, we see that ü is the 129'th character. Or if you take a look here, we see that ü is the 252'nd Unicode character, which is what you get from

>>> ord('ü')
252

So where is the \xc3\xbc coming from and why is it split up into two bytes? and when you decode: b'\xc3\xbc'.decode(), how does it know that these two bytes are for one character?

character-encoding python
1 Answers

On the table you're looking at, you're looking at the section titled "Extended ASCII", more commonly known at ISO/IEC 8859, or latin1. ASCII, as a character set, defines 7-bit characters from 0 to 127. latin1 defines the other 128 single-byte characters and is an extension of ASCII. Python uses UTF-8, which extends ASCII (and hence is compatible with it) but is incompatible with latin1.

The character ü is has Unicode codepoint 0xFC (252 in decimal) and, when using UTF-8, is encoded using two characters.

Lots of online ASCII tables get this wrong. It's inaccurate to call the code points 128 up to 255 ASCII characters, because ASCII doesn't claim to assign any value to those code points.

user_22886590
Related