Why does indexing bytes in Python 3 return an int instead of bytes?

user_1101802 0
Viewed 753

Trying to understand why taking the index of a bytes object returns an int that you can't decode, but a slice returns a bytes object that you can. This seems un-intuitive. When you do the same operation with a string, taking an index at the string position still returns a string.

Working on the Cryptopals challenges, I'm trying to iterate over a byte array to do frequency analysis of an XORed string to count the occurrence of the number of plain text letters. I thought I could do the following, but I'm getting 'int' object has not attribute 'decode' error. From reading the Python docs, that makes sense, a byte array is a mutable sequence of integers, but when testing in the interpreter I was expecting different behavior. 

str_a = bytearray(b'\x1b77316?x\x15\x1b\x7f+x413=x9x(7-6<x7>x:9;76')

for x in str_a:
    _ = x.decode('ascii').upper() 
    if _ in counts:
        counts[_] += 1

If I set a variable to a single byte, I can call decode() on it. I figured I could then iterate over all bytes in a byte string and decode in the same way (hence the loop above). But, since r[0] is an int, that doesn't work. But then if I take r[0:1], it does? I realize I could just call chr(r[0]), but I figured if r.decode() works, r[0].decode() should work too.

>>> r = b'A'
>>> type(r)
<class 'bytes'>
>>> r.decode('ascii')
'A'
>>> r[0:1]
b'A'
>>> r[0:1].decode('ascii')
'A'
>>> type(r[0])
<class 'int'>
>>> r[0]
65
>>> r[0].decode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'decode'

String Example

>>> x = 'AB'
>>> type(x)
<class 'str'>
>>> x[0]
'A'
>>> type(x[0])
<class 'str'>
python-3.x
2 Answers

As you've found out the bytes iterator generates integers not characters. You need to use chr to convert the int value to an str value.

import collections
import string

str_a = b"\x1b77316?x\x15\x1b\x7f+x413=x9x(7-6<x7>x:9;76"
count = collections.defaultdict(int)

for value in str_a:
    value = chr(value).upper()
    if value in string.ascii_uppercase:
        count[value] += 1
user_134912830

The reason that bytes objects iterate as ints is because they are conceptually static arrays of bytes—integers from 0 to 255 (as per PEP 358)—not alternate-encoded strings. Alternate text encodings is a common use case, but reading and writing arbitrary binary data is no less important.

Specifically regarding str.encode and bytes.decode, it doesn't necessarily make sense to be able to call some_bytes[0].decode, because in many encodings, characters may be encoded as multiple bytes. For instance, b'a'.decode('utf-32') fails because UTF-32 uses four bytes per character.

The discussions around PEP 467, which proposed adding, among other things, bytes.iterbytes, might provide additional insight into why bytes behaves how it does.

user_73892640
Related