Different np.std behaviours in pd.apply

Viewed 117

Numpy std function seems to be giving different ddof values in pd.apply depending on whether or not the apply argument is an iterable.

In other words, the line

pd.DataFrame([0, 1]).apply(np.std)

Produces different value from

pd.DataFrame([0, 1]).apply([np.std])

The first one is the correct for numpy with ddof=0 while the second one is the Pandas default with ddof=1.

Why is that the case? What changes from a single argument to a list?

1 Answers

I guess this is because under the covers of pd.DataFrame.apply it uses pd.DataFrame.aggregate if you pass a list-like object, and in aggregate it's replaced with Series.std(), which has the default ddof=1 value. Hence the result:

>>> pd.DataFrame([0, 1]).apply([np.std])
               0
std     0.707107

In the other case when you pass np.std as a regular function, then it doesn't go through any replacement, and np.std has ddof=0 as default, so this becomes:

>>> pd.DataFrame([0, 1]).apply(np.std)
0    0.5
dtype: float64

This is actually a "wontfix" issue: see here. If you want to skip the replacement of numpy functions into pandas functions use it as follows (from the linked github issue):

.groupby(...).agg(lambda x: np.std(x))

Related