Faster way to get fixed percentile on a expanding dataframe

user_1479664 0
Viewed 123

Currently I am doing as below as I need to keep track of particular set of percentiles of expanding set of values.

I was wondering if there is more optimized way to do this using any numpy vectorization tricks


import numpy as np

import pandas as pd

a=np.random.rand(1000)

df=pd.DataFrame(a,columns=['Data'])

val=25

df['25th_Perc']=df.expanding(min_periods=1).apply(lambda x:np.nanpercentile(x,val,interpolation='nearest'),raw=True)
pandas numpy python
1 Answers

This isn't necessarily a NumPy trick but instead more of a Pandas trick.

Instead of using the apply function to apply NumPy's percentile function, you can instead use Pandas' built-in percentile function.

df.Data.expanding(2).quantile(0.25, interpolation="nearest")

This saves your code the effort of extracting the np array and iterating with the apply function and instead directly applies your transform. I suspect that under the hood, it works the same was as your example, but without the overhead of us calling it directly.

This code is much more readable and also faster. Upon testing with your example code, I saw a ~98% speedup but YMMV.

user_149700070
Related