Multiple iterators (using enumerate) for the same iterable, what is going on?

Viewed 130

Consider the following example:

s = 'abc'
[(c1, c2) for j, c2 in enumerate(s) for i, c1 in enumerate(s)]

Output:

[('a', 'a'),
 ('b', 'a'),
 ('c', 'a'),
 ('a', 'b'),
 ('b', 'b'),
 ('c', 'b'),
 ('a', 'c'),
 ('b', 'c'),
 ('c', 'c')]

I would expected the same output if enumerate is called outside the list comprehension and the iterators are assigned to variables:

it1, it2 = enumerate(s), enumerate(s)
[(c1, c2) for j, c2 in it1 for i, c1 in it2]

But I get:

[('a', 'a'), ('b', 'a'), ('c', 'a')]

What is going on? I use Python 3.6.9.

3 Answers

What is happening is that the inner iterator gets exhausted after the first iteration of the outer iterator:

s = 'abc'
it1 = enumerate(s)
it2 = enumerate(s)

for i, x in it1:
    for j, y in it2:  # <-- gets consumed when i = 0 and stays empty
        ...

By contrast:

s = 'abc'

for i, x in enumerate(s):
    for j, y in enumerate(s):  # <-- gets recreated at each iteration
        ....

If you need persistence, enclose it in a list or tuple:

itr = list(enumerate(s))
print([(c1, c2) for j, c2 in itr for i, c1 in itr])
# [('a', 'a'), ('b', 'a'), ('c', 'a'), ('a', 'b'), ('b', 'b'), ('c', 'b'), ('a', 'c'), ('b', 'c'), ('c', 'c')]

although note the different memory footprint of using enumerate() multiple times versus having it enclosed in a list or tuple.

The difference is that in:

s = 'abc'
[(c1, c2) for j, c2 in enumerate(s) for i, c1 in enumerate(s)]

A new c1 enumerator is created for each value yielded on the first for. While on your second example, the same enumerator is used (it2) - and it gets exausted once it reaches "c" - when the first for advances to the next iteration (c2 = "b") and tries to iterate "it2", it is already exhausted - and the whole expression ends.

enumerate build-in function returns an iterator, once the elements are exhausted it stops

in your first version, you are building a new enumerate(s) for each iteration of the first loop and in the second loop you are consuming it,

in your second version, it2 has finished his elements from the first iteration of the first loop

s = 'abc'

def my_enumerate(s, name):
    print('a new enumerate: ', name)
    for i in enumerate(s):
        yield i
    print(f'the enumerate {name} is exhausted ')

[(c1, c2) for j, c2 in my_enumerate(s, 'it1') for i, c1 in my_enumerate(s, 'it2')]

output:

a new enumerate:  it1
a new enumerate:  it2
the enumerate it2 is exhausted 
a new enumerate:  it2
the enumerate it2 is exhausted 
a new enumerate:  it2
the enumerate it2 is exhausted 
the enumerate it1 is exhausted 
[('a', 'a'),
 ('b', 'a'),
 ('c', 'a'),
 ('a', 'b'),
 ('b', 'b'),
 ('c', 'b'),
 ('a', 'c'),
 ('b', 'c'),
 ('c', 'c')]

and for:

it1, it2 = my_enumerate(s, 'it1'), my_enumerate(s, 'it2')
[(c1, c2) for j, c2 in it1 for i, c1 in it2]

output:

a new enumerate:  it1
a new enumerate:  it2
the enumerate it2 is exhausted 
the enumerate it1 is exhausted 
[('a', 'a'), ('b', 'a'), ('c', 'a')]
Related