Pandas DataFrame.apply going very slow for scipy.stats

user_2368475 0
Viewed 171

I am trying to create a custom DataFrame.describe() function that will return

  • numpy statistics
  • scipy statistics
  • quantiles from some source

This should reproduce my issue

# Python 3.6.6, Pandas 0.22.0

import pandas as pd
import numpy as np
import scipy.stats as sc

d = pd.DataFrame(np.random.randint(0,10, size=100000))
d.apply(np.mean) #Works
d.apply(np.std) #Works
d.apply(sc.kurtosis) #Works
d.apply([np.mean, np.std]) #Works
d.apply([np.mean, sc.kurtosis]) # Gets stuck

It runs forever if I include the scipy functions, but finishes very quickly if I only use numpy. If there is a better way to mimic the output of pd.DataFrame.describe() with my own custom list of statistical output I am open to that too.

pandas scipy dataframe numpy python
1 Answers

pd.DataFrame.apply isn't magic. It's just a Python-level loop and a convenience method. Except here, it doesn't add much convenience. It doesn't accept lists either, as it applies a single function along an axis, so your code is erroneous.

You can feed your dataframe as an argument to all these functions directly, and this exhibits consistent performance:

# Python 3.6.0, Pandas 0.19.2

import pandas as pd
import numpy as np
import scipy as sc

np.random.seed(0)
d = pd.DataFrame(np.random.randint(0,10, size=10**6))

%timeit np.mean(d)                                          # 1.3 ms per loop
%timeit np.std(d)                                           # 2.82 ms per loop
%timeit sc.stats.kurtosis(d)                                # 33 ms per loop
%timeit [func(d) for func in (np.mean, np.std)]             # 3.95 ms per loop
%timeit [func(d) for func in (np.mean, sc.stats.kurtosis)]  # 34.8 ms per loop
user_92095460
Related