Why does a dictionary count in some cases faster than collections.Counter?

Question

I needed a solution for extracting unique elements from a non-unique list along with counting its duplicate elements.

The purpose of the solution was to use it in an algorithm for creating unique combinations from a non-unique list. The list sizes from which combinations are created in such case are usually very small (less than 50 elements), but my goal was to find the overall fastest code trying to optimize whenever and wherever possible (even if gaining only very tiny amount of running time).

Pythons collections module provides an exactly for such purpose suited specialized collections.Counter, but there are apparently cases in which usage of a simple dictionary instead of collections.Counter leads to faster code like you can check out yourself using the code below:

from time   import time   as t
from timeit import timeit as tt

from collections import Counter
def counter(iterable):
    dctCounter = {}
    for item in iterable:
        if item in dctCounter: 
            dctCounter[item] += 1
        else: 
            dctCounter[item]  = 1
    return dctCounter

for n, N in [(1,10), (10,1), (1,50), (50,1), (1,100), (100,1), (1,200), (200, 1), (1, 500), (500, 1), (1, 1000), (1000,1)]:
    lstItems = n*list(range(N))
    for noLoops in [10**p for p in range(5, 6)]: 
        s = t()
        for _ in range(noLoops): 
            dctCounter = counter(lstItems)
        e = t()
        timeDctFctn = e - s
        s = t()
        for _ in range(noLoops): 
            objCounter = Counter(lstItems)
        e = t()
        timeCollCtr = e - s
        timeitCollCtr = tt("objCounter=Counter(lstItems)", "from __main__ import Counter, lstItems", number=noLoops)
        timeitDctFctn = tt("dctCounter=counter(lstItems)", "from __main__ import counter, lstItems", number=noLoops)
        # print("Loops: {:7}, time/timeit CollCtr: {:7.5f}/{:7.5f} DctFctn: {:7.5f}/{:7.5f} sec. lstSize: {:3}, %uniq: {:3.0f}".format(noLoops, timeCollCtr, timeitCollCtr, timeDctFctn, timeitDctFctn, n*N, 100.0/n))
        print("collections.Counter(): {:7.5f},  def counter(): {:7.5f} sec. lstSize: {:3}, %uniq: {:3.0f}, ({} timitLoops)".format(timeitCollCtr, timeitDctFctn, n*N, 100.0/n, noLoops))    
        # print('-----------------------------------------------------------------------------------------------------------')

Here the output:

python3.6 -u "collections.Counter-vs-dictionaryAsCounter_Cg.py"
collections.Counter(): 0.36461, def counter(): 0.09592 sec. lstSize:  10, %uniq: 100, (100000 timitLoops)
collections.Counter(): 0.36444, def counter(): 0.12286 sec. lstSize:  10, %uniq:  10, (100000 timitLoops)
collections.Counter(): 0.58627, def counter(): 0.43233 sec. lstSize:  50, %uniq: 100, (100000 timitLoops)
collections.Counter(): 0.52399, def counter(): 0.54106 sec. lstSize:  50, %uniq:   2, (100000 timitLoops)
collections.Counter(): 0.82332, def counter(): 0.81436 sec. lstSize: 100, %uniq: 100, (100000 timitLoops)
collections.Counter(): 0.72513, def counter(): 1.06823 sec. lstSize: 100, %uniq:   1, (100000 timitLoops)
collections.Counter(): 1.27130, def counter(): 1.59476 sec. lstSize: 200, %uniq: 100, (100000 timitLoops)
collections.Counter(): 1.13817, def counter(): 2.14566 sec. lstSize: 200, %uniq:   0, (100000 timitLoops)
collections.Counter(): 3.16287, def counter(): 4.26738 sec. lstSize: 500, %uniq: 100, (100000 timitLoops)
collections.Counter(): 2.64247, def counter(): 5.67448 sec. lstSize: 500, %uniq:   0, (100000 timitLoops)
collections.Counter(): 4.89153, def counter(): 7.68661 sec. lstSize:1000, %uniq: 100, (100000 timitLoops)
collections.Counter(): 6.06389, def counter():13.92613 sec. lstSize:1000, %uniq:   0, (100000 timitLoops)
>Exit code: 0

P.S. It seems that collections.Counter() fails to be up to the expectations also in another context as this above. See here: https://stackoverflow.com/questions/41594940/why-is-collections-counter-so-slow

Why does a dictionary count in some cases faster than collections.Counter?

1 Answers