Assign iterators to a Python slice

user_4119680 0
Viewed 231

Are iterators consumed before being assigned to a slice in Python? By "consumed before being assigned" I mean the elements are created in the memory all at the same time (put into list or tuple) before the slice assignment happens. The other approach would be to put the elements from the iterator in the slice one-by-one, so elements are not created in the memory all at the same time.

For example, let`s consider this code:

from itertools import islice
from heapq import merge

c = [0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5] + list(range(10))
lo, mid, hi = 0, 10, 20
c[lo:hi] = merge(islice(iter(c), lo, mid), islice(iter(c), mid, hi))

merge returns an iterator that picks the smallest element of the two iterators given to merge. Would these iterator be consumed before the slice assignment happens?

slice iterator python
2 Answers

This depends on the implementation of the sequence that is being slice-assigned into. The sequence receives the iterator directly, and the details of consuming the iterator are up to the sequence.

For lists, the current CPython implementation consumes the iterator up front before any modification to the list:

v_as_SF = PySequence_Fast(v, "can only assign an iterable");

PySequence_Fast will build a list out of any argument that isn't already a list or tuple.

There's also self-assignment handling, because turning the input into a list isn't enough to be safe when you're slice-assigning a list into itself:

/* Special case "a[i:j] = a" -- copy b first */
v = list_slice(b, 0, Py_SIZE(b));

I don't think any of this list behavior is documented.

user_23571120

UPDATED ANSWER:

Yes, the given iterables consumes before passing to slice():

For empirical proof a simple example with memory approx. consuming:

1.) An example code with 10e6 elements of array.

from itertools import islice
from heapq import merge

c = [0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5] + list(range(int(10e6)))
lo, mid, hi = 0, 10, 20

c[lo:hi] = merge(islice(iter(c), lo, mid), islice(iter(c), mid, hi))

There's no noticable time and memory consumption.

2.) The same code with 10e7 elements of array.

c = [0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5] + list(range(int(10e7)))

meant 30% of total memory consumption of the same configuration.

3.) The same code with 10e8 elements of array.

Ends up in a 100%+ memory need of course.

Note that the volume of slicing remained the same, but the memory consumption increased significantly.

user_45920590
Related