Performance curiosity on pandas series `any`, `max`, `sum` vs python builtins

user_14094460 0
Viewed 163

Seeking performance in python applications using pandas/numpy often benefits from the use of the pandas/numpy implemented methods other than own implemented code such as through looping. This might be a bad introduction to the question I have, but in the following screenshot (if I hadn't tested it) I was expecting the versions using the series' methods to run faster than the python builtins. Since that's not the case, it means I built a false intuition on this example, but I could not yet find the reason for this. So the question is, why in this case the use of the python builtins has higher performance than the methods applied on the series (am I missing something else?)?

enter image description here

pandas built-in numpy python
2 Answers

Pandas has its own functions which are way different than Python's built in functions, therefore if you call Series.max() you are in fact calling nanops._nanminmax() which is added via the IndexOpsMixin instead of builtins.max()

Each behave differently, thus have different performance times.

Similarly for the rest of the methods. If you are curious, check the source code for Series class and classes it inherits from for the exact differences between builtin functions and Pandas' implementation.

user_59940410

am I missing something else?

You assumed that you always will get same result, which is not true. Example showing different output for sum and pandas.Series.sum is

import pandas as pd
s = pd.Series([1.0,2.0,float("nan")])
print(s.sum())
print(sum(s))

output

3.0
nan
user_107859750
Related