Python bytes object from generator

bytes(iterator) create bytes object from iterator using internal C-API _PyBytes_FromIterator function, which use special _PyBytes_Writer protocol. It internaly use a buffer, which resize when it overflows using a rule:

bufsize += bufsize  / OVERALLOCATE_FACTOR

For linux OVERALLOCATE_FACTOR=4, for windows OVERALLOCATE_FACTOR=2.

Those. This process looks like writing to a file in RAM. At the end, the contents of the buffer returns.

this is probably more a remark or discussion start than an answer but I think it's better to format it like this. I just hooking in 'cause I find this topic very inteeresting as well.

I would recommend to paste the real call and generator mock. Since imho the generator expression example does not fit really well for your question. And the example code you pasted does not work.

Normally you have a generator like this: (in your case calling the module instead of generating numbers of course ..) Example slightly modified from dabeaz Update: deleted the explicit byte creatinon.

def genbytes():
    for i in range(100):
         yield i**2

You would the probably call it with something like this:

for newbyte in genbytes():
    print(newbyte)
    print(id(newbyte))
    input("press key...")

leeds to:

Do I have to assume that for every element the generator yields, there is a new bytes object created

I would say absolutely yes. That's the way that yield works. And what we can see above. Bytes always has a new id.

Normally this shouldn't be a problem since you want to consume the bytes one by one and then collect it into something like you suggested a bytearray using bytearray append

But even if new bytes are produced I think this is at least not producing 33 MB in the input at once and returning them.

I add this exerpt from PEP 289 where the equivalence of a gen expression and the generator in "function" style is pointed out:

The semantics of a generator expression are equivalent to creating an anonymous generator function and calling it. For example:

g = (x**2 for x in range(10))
print g.next()

is equivalent to:

def __gen(exp):
    for x in exp:
        yield x**2
g = __gen(iter(range(10)))
print g.next()

So bytes(gen) also calls gen.next() (which yields ints in this case but probably bytes in your real case) while itrerating over the gen.

Creating immutable types from arbitrary sized iterables is a common task for Python. Most notably, tuple is an ubiquitous immutable type. This makes it a necessity to efficiently create such instances – constantly creating and discarding intermediate instances would not be efficient.

Using a generator as the source of bytes will be reasonably fast. However, directly providing data as a fixed tuple or list would be faster, and a readable buffer can even be directly copied to a new bytes object. If the data is provided as a generator, intermediate tuple, list or bytearray will not offer a benefit.

The general task of creating a fixed size instance from an iterable is not unique to immutable types. Even a mutable type such as list has a fixed size at any specific point in time. While filling a list by .appending elements during iteration, Python cannot know the final size ahead of time.
Looking at the size of a list in such a scenario reveals the general strategy of turning an arbitrary size iterable into a specific sized container:

>>> items = []
>>> for i in (i*2 for i in range(10)):
>>>     items.append(i)
>>>     print(i // 2, "\t", sys.getsizeof(items))
0    88
1    88
2    88
3    88
4    120
5    120
6    120
7    120
8    184
9    184

As can be seen, Python does not grow the list for each item. Instead, when the list is too small an entire chunk of space is added that can hold several items before resizing is required again.

A naive implementation of bytes when presented with an arbitrary sized iterable would first convert its argument to a list, then copy the content to a new bytes instance. This would already create only one bytes instance and one temporary, auxiliary list – much more efficient than creating a new instance per element.

However, it is possible to go one step further: bytes is an immutable sequence of immutable integers, so it does not have to preserve the identity of its items: since the value of each bytes item fits exactly into a byte / C char, the items can be written directly to memory.
In the CPython 3.9 implementation, constructing bytes from iterables has some special cases for direct access buffers, guaranteed-size list and tuple, and a generic path for any other iterable that is not a string. The case of a generator also falls into the latter path.

This generic path uses a struct as a primitive, low-level "list" for chars – it uses a char array for storage, and holds metadata like current and maximum size. The byte value of wach item fetched from the iterable/generator is simply copied directly to the current end of the char array; if the char array becomes too small, it is replaced by a new, larger char array. Once the iterable/generator is exhausted, the char array can be directly used as the internal storage of the new bytes object.

When looking at the timings, the important part is whether the data already comes from a generator, or whether one can chose other formats.

When there already is a generator, feeding it directly to bytes is the fastest – though not by much.

%timeit bytes(i*2 % 256 for i in range(2**20))
107 ms ± 411 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit bytes(bytearray(i*2 % 256 for i in range(2**20)))
111 ms ± 946 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit bytes(tuple(i*2 % 256 for i in range(2**20)))
114 ms ± 1.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit bytes(list(i*2 % 256 for i in range(2**20)))
115 ms ± 741 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

However, if the initial data structure can be freely chosen, a list is faster at the cost of intermediate storage.

%timeit bytes([i*2 % 256 for i in range(2**20)])
94.5 ms ± 1.11 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

When directly creating the data in a C/Cython module, providing a bytes/bytesarray is by far fastest and has low overhead.