How to decode longest sub-bytes into str?

Question

Suppose I read a long bytes object from somewhere, knowing it is utf-8 encoded. But the read may not fully consume the available content so that the last character in the stream may be incomplete. Calling bytes.decode() on this object may result in a decode error. But what really fails is only the last few bytes. Is there a function that works in this case, returning the longest decoded string and the remaining bytes?

utf-8 encodes a character into at most 4 bytes, so trying to decode truncated bytes should work, but a vast majority of computation will be wasted, and I don't really like this solution.

To give a simple but concrete example:

>>> b0 = b'\xc3\x84\xc3\x96\xc3'
>>> b1 = b'\x9c\xc3\x84\xc3\x96\xc3\x9c'
>>> (b0 + b1).decode()
>>> 'ÄÖÜÄÖÜ'

(b0 + b1).decode() is fine, but b0.decode() will raise. The solution should be able to decode b0 for as much as possible and return the bytes that cannot be decoded.

user_674039 · Answer

You are describing the basic usage of io.TextIOWrapper: a buffered text stream over a binary stream.

>>> import io 
>>> txt = 'before\N{PILE OF POO}after' 
>>> b = io.BytesIO(txt.encode('utf-8'))
>>> t = io.TextIOWrapper(b) 
>>> t.read(5) 
'befor'
>>> t.read(1) 
'e'
>>> t.read(1)
''
>>> t.read(1) 
'a'

Contrast with reading a bytes stream directly, where it would be possible to read halfway through an encoded pile of poo:

>>> b.seek(0) 
0
>>> b.read(5)
b'befor'
>>> b.read(1)
b'e'
>>> b.read(1)
b'\xf0'
>>> b.read(1)
b'\x9f'
>>> b.read(1)
b'\x92'
>>> b.read(1)
b'\xa9'
>>> b.read(1)
b'a'

Specify encoding="utf-8" if you want to be explicit. The default encoding, i.e. locale.getpreferredencoding(False), would usually be utf-8 anyway.

user_355230 · Answer

As I mentioned in the comments under @wim's answer, I think you could use the codecs.iterdecode() incremental decoder to do this. Since it's a generator function, there's no need to manually save and restore its state between iterative calls to it.

Here's how how it might be used to handle a situation like the one you described:

import codecs
from random import randint


def reader(sequence):
    """ Yield random length chunks of sequence until exhausted. """

    plural = lambda word, n, ending='s': (word+ending) if n > 1 else word

    i = 0
    while i < len(sequence):
        size = randint(1, 4)
        chunk = sequence[i: i+size]

        hexrepr = '0x' + ''.join('%02X' % b for b in chunk)
        print('read {} {}: {}'.format(size, plural('byte', len(chunk)), hexrepr))

        yield chunk
        i += size


bytes_obj = b'\xc3\x84\xc3\x96\xc3\x9c\xc3\x84\xc3\x96\xc3\x9c'  # 'ÄÖÜÄÖÜ'

for decoded in codecs.iterdecode(reader(bytes_obj), 'utf-8'):
    print(decoded)

Sample output:

read 3 bytes: 0xC384C3
Ä
read 1 byte: 0x96
Ö
read 1 byte: 0xC3
read 3 bytes: 0x9CC384
ÜÄ
read 2 bytes: 0xC396
Ö
read 4 bytes: 0xC39C
Ü

How to decode longest sub-bytes into str?

2 Answers