Why is pandas.dataframe.groupby faster when assigned to variable first?

user_9010605 0
Viewed 221

Can anyone please help understand why the following two ways of doing, what i thought was, otherwise the same thing with the pandas.dataframe groupby method, complete in such different times according to iPython's Magic %timeit?

%timeit somedf.groupby('someBoolColumn')['someBoolColumn'].count()
484 µs ± 9.52 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit grp = somedf.groupby('someBoolColumn')
grp['someBoolColumn'].count()
146 µs ± 1.47 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

somedf has 7200 rows and 24 columns.

I cannot find,

  1. why the two liner assigning the object to variable grp first is >3
    times faster, nor;

  2. if this is just specific to the groupby method or something more general to pandas or even python, for example, about variable assigment.

Many thanks for any enlightenment as this would really help with many much larger dataframes i'd like to process, repeatedly with many different combinations of parameters.

pandas optimization arrays python performance
1 Answers

Ipython's %timeit docs state:

In cell mode, the statement in the first line is used as setup code (executed but not timed) and the body of the cell is timed. The cell body has access to any variables created in the setup code.

(my emphasis). cell mode is triggered by using the double-percent form of %%timeit. There is also a blurb in the documentation that IPython prints when you type %magic at the IPython prompt:

%%timeit x = numpy.random.randn((100, 100))
numpy.linalg.svd(x)

will time the execution of the numpy svd routine, running the assignment of x as part of the setup phase, which is not timed.

Thus, 

%%timeit grp = somedf.groupby('someBoolColumn')
grp['someBoolColumn'].count()

is timing grp['someBoolColumn'].count(), but not the assignment grp = somedf.groupby('someBoolColumn').

How to use %%timeit without a setup line:

To use %%timeit to time both statements, simply leave the first line blank after %%timeit:

%%timeit 
grp = somedf.groupby('someBoolColumn')
grp['someBoolColumn'].count()

The cell is finished by typing Enter twice.

user_1905970
Related