itertools.tee not making copies of groupby iterator

user_9349697 0
Viewed 72

I just learnt that itertools.tee() can be used to make any number of copies of a given iterator.

So I am trying to create two copies of groupby and loop over each one of them. However, the second iterator returns no groups.

import itertools
l = [("a", 1), ("a", 2), ("b", 3), ("b", 4)]
groups = itertools.groupby(l, lambda x: x[0])

groups1, groups2 = itertools.tee(groups, 2)

for key, group in groups1:
    print(key + ": " + str(list(group)))

for key, group in groups2:
    print(key + ": " + str(list(group)))

Also, I found this in the docs and I couldn't exactly understand when tee() is not faster than list()

This itertool may require significant auxiliary storage (depending on how much temporary data needs to be stored). In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee().

Any help would be greatly appreciated. Thank you!

itertools python
1 Answers

groupby is a nested iterator; the "outer" returns a sequence of _grouper objects, each of which is also an iterator. You tee the outer one successfully, but the inner iterators aren't copied so they get exhausted during the first loop. If you'll notice, your second loop does print out the keys for each group, it's just the groups themselves that are empty.

Separately, about list vs tee -- what tee does is fetch each item from the underlying iterator and save n copies of it. As each independent iterator advances, its copy of a particular item can get discarded. If you advance all your copies in lock-step, this is fine. But if you e.g. fully exhaust one copy without touching any of the others, you've essentially just created a n full copies of the original iterator.

As the documentation states:

In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee().

user_2574650
Related