So, for those who would like to know the summary of that discussion. The final top scores for counting a 50 million-lengthed generator expression using:

• len(list(gen)),
• len([_ for _ in gen]),
• sum(1 for _ in gen),
• ilen(gen) (from more_itertool),
• reduce(lambda c, i: c + 1, gen, 0),

sorted by performance of execution (including memory consumption), will make you surprised:

```

1: test_list.py:8: 0.492 KiB

gen = (i for i in data*1000); t0 = monotonic(); len(list(gen))

('list, sec', 1.9684218849870376)

2: test_list_compr.py:8: 0.867 KiB

gen = (i for i in data*1000); t0 = monotonic(); len([i for i in gen])

('list_compr, sec', 2.5885991149989422)

3: test_sum.py:8: 0.859 KiB

gen = (i for i in data*1000); t0 = monotonic(); sum(1 for i in gen); t1 = monotonic()

('sum, sec', 3.441088170016883)

4: more_itertools/more.py:413: 1.266 KiB

d = deque(enumerate(iterable, 1), maxlen=1)

test_ilen.py:10: 0.875 KiB
gen = (i for i in data*1000); t0 = monotonic(); ilen(gen)

('ilen, sec', 9.812256851990242)

5: test_reduce.py:8: 0.859 KiB

gen = (i for i in data*1000); t0 = monotonic(); reduce(lambda counter, i: counter + 1, gen, 0)

('reduce, sec', 13.436614598002052) ```

So, len(list(gen)) is the most frequent and less memory consumable

I thought it could be worthwhile to have a micro-benchmark comparing the run-times of the different approaches mentioned here.

Disclaimer: I'm using simple_benchmark (a library written by me) for the benchmarks and also include iteration_utilities.count_items (a function in a third-party-library written by me).

To provide a more differentiated result I've done two benchmarks, one only including the approaches that don't build an intermediate container just to throw it away and one including these:

from simple_benchmark import BenchmarkBuilder
import more_itertools as mi
import iteration_utilities as iu

b1 = BenchmarkBuilder()
b2 = BenchmarkBuilder()

@b1.add_function()
@b2.add_function()
def summation(it):
    return sum(1 for _ in it)

@b1.add_function()
def len_list(it):
    return len(list(it))

@b1.add_function()
def len_listcomp(it):
    return len([_ for _ in it])

@b1.add_function()
@b2.add_function()
def more_itertools_ilen(it):
    return mi.ilen(it)

@b1.add_function()
@b2.add_function()
def iteration_utilities_count_items(it):
    return iu.count_items(it)

@b1.add_arguments('length')
@b2.add_arguments('length')
def argument_provider():
    for exp in range(2, 18):
        size = 2**exp
        yield size, [0]*size

r1 = b1.run()
r2 = b2.run()

import matplotlib.pyplot as plt

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=[15, 18])
r1.plot(ax=ax2)
r2.plot(ax=ax1)
plt.savefig('result.png')

The results were:

It uses log-log-axis so that all ranges (small values, large values) can be inspected. Since the plots are intended for qualitative comparison the actual values aren't too interesting. In general the y-axis (vertical) represents the time and the x-axis (horizontal) represents the number of elements in the input "iterable". Lower on the vertical axis means faster.

The upper plot shows the approaches where no intermediate list was used. Which shows that the iteration_utilities approach was fastest, followed by more_itertools and the slowest was using sum(1 for _ in iterator).

The lower plot also included the approaches that used len() on an intermediate list, once with list and once with a list comprehension. The approach with len(list) was fastest here, but the difference to the iteration_utilities approach is almost negligible. The approach using the comprehension was significantly slower than using list directly.

Summary

Any approach mentioned here did show a dependency on the length of the input and iterated over ever element in the iterable. There is no way to get the length without the iteration (even if the iteration is hidden).

If you don't want third-party extensions then using len(list(iterable)) is definitely the fastest approach of the tested approaches, it however generates an intermediate list which could use significant more memory.

If you don't mind additional packages then iteration_utilities.count_items would be almost as fast as the len(list(...)) function but doesn't require additional memory.

However it's important to note that the micro-benchmark used a list as input. The result of the benchmark could be different depending on the iterable you want to get the length of. I also tested with range and a simple genertor-expression and the trends were very similar, however I cannot exclude that the timing won't change depending on the type of input.

One simple way is using set() built-in function:

iter = zip([1,2,3],['a','b','c'])
print(len(set(iter)) # set(iter) = {(1, 'a'), (2, 'b'), (3, 'c')}
Out[45]: 3

or

iter = range(1,10)
print(len(set(iter)) # set(iter) = {1, 2, 3, 4, 5, 6, 7, 8, 9}
Out[47]: 9

This is theoretically impossible: this is, in fact, the Halting Problem.

Proof

Assume in contradiction it were possible to determine the length (or infinite length) of any generator g, using a function len(g).

For any program P, let us now convert P into a generator g(P): For every return or exit point in P, yield a value instead of returning it.

If len(g(P)) == infinity, P doesn't stop.

This solves the Halting Problem, which is known to be impossible, see Wikipedia. Contradiction.

Thus, it is impossible to count the elements of a generic generator without iterating over it (==actually running through the program).

More concretely, consider

def g():
    while True:
        yield "more?"

The length is infinite. There are infinitely many such generators.

This is against the very definition of an iterator, which is a pointer to an object, plus information about how to get to the next object.

An iterator does not know how many more times it will be able to iterate until terminating. This could be infinite, so infinity might be your answer.

Presumably, you want count the number of items without iterating through, so that the iterator is not exhausted, and you use it again later. This is possible with copy or deepcopy

import copy

def get_iter_len(iterator):
    return sum(1 for _ in copy.copy(iterator))

###############################################

iterator = range(0, 10)
print(get_iter_len(iterator))

if len(tuple(iterator)) > 1:
    print("Finding the length did not exhaust the iterator!")
else:
    print("oh no! it's all gone")

The output is "Finding the length did not exhaust the iterator!"

Optionally (and unadvisedly), you can shadow the built-in len function as follows:

import copy

def len(obj, *, len=len):
    try:
        if hasattr(obj, "__len__"):
            r = len(obj)
        elif hasattr(obj, "__next__"):
            r = sum(1 for _ in copy.copy(obj))
        else:
            r = len(obj)
    finally:
        pass
    return r