Length of generator output

Admin 1
Viewed 114546

Python provides a nice method for getting length of an eager iterable, len(x) that is. But I couldn't find anything similar for lazy iterables represented by generator comprehensions and functions. Of course, it is not hard to write something like:

def iterlen(x):
  n = 0
  try:
    while True:
      next(x)
      n += 1
  except StopIteration: pass
  return n

But I can't get rid of a feeling that I'm reimplementing a bicycle.

(While I was typing the function, a thought struck my mind: maybe there really is no such function, because it "destroys" its argument. Not an issue for my case, though).

P.S.: concerning the first answers - yes, something like len(list(x)) would work too, but that drastically increases the usage of memory.

P.P.S.: re-checked... Disregard the P.S., seems I made a mistake while trying that, it works fine. Sorry for the trouble.

iterable generator python
9 Answers

For those who would like to know the summary of that discussion. The final top scores for counting a 50 million-lengthed generator expression using:

  • len(list(gen)),
  • len([_ for _ in gen]),
  • sum(1 for _ in gen),
  • ilen(gen) (from more_itertools),
  • reduce(lambda c, i: c + 1, gen, 0),

sorted by performance of execution (including memory consumption), will make you surprised:

#1: test_list.py:8: 0.492 KiB
    gen = (i for i in data*1000); t0 = monotonic(); len(list(gen))
('list, sec', 1.9684218849870376)

#2: test_list_compr.py:8: 0.867 KiB
    gen = (i for i in data*1000); t0 = monotonic(); len([i for i in gen])
('list_compr, sec', 2.5885991149989422)

#3: test_sum.py:8: 0.859 KiB
    gen = (i for i in data*1000); t0 = monotonic(); sum(1 for i in gen); t1 = monotonic()
('sum, sec', 3.441088170016883)

#4: more_itertools/more.py:413: 1.266 KiB
    d = deque(enumerate(iterable, 1), maxlen=1)
   
    test_ilen.py:10: 0.875 KiB
    gen = (i for i in data*1000); t0 = monotonic(); ilen(gen)
('ilen, sec', 9.812256851990242)

#5: test_reduce.py:8: 0.859 KiB
    gen = (i for i in data*1000); t0 = monotonic(); reduce(lambda counter, i: counter + 1, gen, 0)
('reduce, sec', 13.436614598002052)

So, len(list(gen)) is the most frequent and less memory consumable

user_41951720

There isn't one because you can't do it in the general case - what if you have a lazy infinite generator? For example:

def fib():
    a, b = 0, 1
    while True:
        a, b = b, a + b
        yield a

This never terminates but will generate the Fibonacci numbers. You can get as many Fibonacci numbers as you want by calling next().

If you really need to know the number of items there are, then you can't iterate through them linearly one time anyway, so just use a different data structure such as a regular list.

Adam_Rosenfield_95300

You can use enumerate() to loop through the generated data stream, then return the last number -- the number of items.

I tried to use itertools.count() with itertools.izip() but no luck. This is the best/shortest answer I've come up with:

#!/usr/bin/python

import itertools

def func():
    for i in 'yummy beer':
        yield i

def icount(ifunc):
    size = -1 # for the case of an empty iterator
    for size, _ in enumerate(ifunc()):
        pass
    return size + 1

print list(func())
print 'icount', icount(func)

# ['y', 'u', 'm', 'm', 'y', ' ', 'b', 'e', 'e', 'r']
# icount 10

Kamil Kisiel's solution is way better:

def count_iterable(i):
    return sum(1 for e in i)
Admin1
Related