Why is pandas ewm passed with times so slow?

user_9929376 0
Viewed 1010

Suppose I have the following data:

import pandas as pd
import numpy as np
import datetime as dt

idx = pd.date_range("2010/01/01", "2020/01/01", freq='1T')
n = len(idx)

data = pd.DataFrame({'A': np.random.random(n), 'B': np.random.random(n), 'C': np.random.random(n)}, index=idx)

I can very quickly calculate the exponential moving average of this with halflife 1 hour with:

data.ewm(halflife=60).mean()

However, if I try:

data.ewm(halflife=dt.timedelta(hours=1), times=data.index).mean()

It is very slow (to the point of exiting the code). Why is this?

pandas moving-average numpy python
1 Answers

I have noted the same thing, don't know why. The timedelta method is about 4000 times slower on my laptop. The two methods will not produce the same result if the time steps are not uniform though, see below

import pandas as pd
import numpy as np
import datetime as dt
import time

idx = pd.date_range("2019/12/24", "2020/01/01", freq='1T')
n = len(idx)
rand_hours = pd.to_timedelta(np.random.random(n) / 3.0, unit='h')
idx += rand_hours

data = pd.DataFrame({'A': np.random.random(n), 'B': np.random.random(n), 'C': np.random.random(n)}, index=idx)

start = time.process_time()
v1 = data.ewm(halflife=60).mean()
t1 = time.process_time() - start
print('T1', t1)

start = time.process_time()
v2 = data.ewm(halflife=dt.timedelta(hours=1), times=data.index).mean()
t2 = time.process_time() - start
print('T2', t2, 'scale', t2 / t1)

print(0.5 * (v1 - v2) / (v1 + v2))

Produces

T1 0.001174116999999919
T2 4.715054932 scale 4015.830562031148
                                      A         B         C
2019-12-24 00:19:07.834616400  0.000000  0.000000  0.000000
2019-12-24 00:07:25.226215200 -0.005825  0.017740  0.000962
2019-12-24 00:16:53.113740000 -0.003800  0.008667  0.000355
2019-12-24 00:15:56.813227200 -0.002508  0.006256  0.000556
2019-12-24 00:04:50.909022000 -0.006851  0.007318 -0.000670
...                                 ...       ...       ...
2020-01-01 00:04:22.018974000 -0.000450 -0.000508  0.001395
2020-01-01 00:10:42.774960000 -0.000348 -0.000437  0.001404
2020-01-01 00:13:37.267552799 -0.000231 -0.000319  0.001293
2020-01-01 00:09:07.053290400 -0.000228 -0.000314  0.001287
2020-01-01 00:07:54.683781599 -0.000180 -0.000353  0.001279

[11521 rows x 3 columns]
user_9739560
Related