I have experienced a strange issue in python (2.7.6.2) / pandas (0.13, also in 0.18), when applying a formula on a dataframe. Apparently the result of the formula is different when applying with map/lambda, instead of applying directly on the numbers. It seems to be a bug for me, and I would be curious what the reason is respectively how to avoid such issue.
...
I now prepared the case in a way which allows to reproduce it, and this makes the problem somewhat clearer:
data15min = [ 5.4753, 5.4863, 5.2497, 5.057, 5.0917, 5.3467, 5.7513, 5.6, 5.342 ]
index = pd.date_range("2015-10-17 18:00:00", periods=9, freq='15T')
columns = ['v03']
df15 = pd.DataFrame(data15min, index=index, columns=columns)
df_h = df15.rolling(min_periods=4,window=4,center=False).mean()
df_m = df_h['v03'].map(lambda x: np.nan if np.isnan(x) else int(x*100.))
The last value from df_h was the value which was calculated wrongly. The value itself looks fine (the mean of 5.3467, 5.7513, 5.6, 5.342 is exactly 5.51):
In [99]: df_h
Out[99]:
v03
2015-10-17 18:00:00 NaN
2015-10-17 18:15:00 NaN
2015-10-17 18:30:00 NaN
2015-10-17 18:45:00 5.317075
2015-10-17 19:00:00 5.221175
2015-10-17 19:15:00 5.186275
2015-10-17 19:30:00 5.311675
2015-10-17 19:45:00 5.447425
2015-10-17 20:00:00 5.510000
After applying the formula with map, I get however 550:
In [100]: df_m
Out[100]:
2015-10-17 18:00:00 NaN
2015-10-17 18:15:00 NaN
2015-10-17 18:30:00 NaN
2015-10-17 18:45:00 531.0
2015-10-17 19:00:00 522.0
2015-10-17 19:15:00 518.0
2015-10-17 19:30:00 531.0
2015-10-17 19:45:00 544.0
2015-10-17 20:00:00 550.0
Freq: 15T, Name: v03, dtype: float64
I assumed this is due to inaccurate number representation, but I get different behaviour when applying the formula directly on the numbers:
In [103]: int(np.mean([5.3467, 5.7513, 5.6, 5.342])*100.)
Out[103]: 551
To make the confusion complete, I also get also a different result with map when having a slightly shorter dataframe with the same relevant values:
data15min = [ 5.3467, 5.7513, 5.6, 5.342 ]
index = pd.date_range("2015-10-17 19:15:00", periods=4, freq='15T')
columns = ['v03']
df15 = pd.DataFrame(data15min, index=index, columns=columns)
df_h = df15.rolling(min_periods=4,window=4,center=False).mean()
df_m = df_h['v03'].map(lambda x: np.nan if np.isnan(x) else int(x*100.))
In [104]: df_m
Out[104]:
2015-10-17 19:15:00 NaN
2015-10-17 19:30:00 NaN
2015-10-17 19:45:00 NaN
2015-10-17 20:00:00 551.0
Freq: 15T, Name: v03, dtype: float64
I am confused, and concerned about getting wrong results. If this is connected to inaccurate internal number representation (which would be surprising if this issue would behave different for the shown cases), I really wonder how I could avoid getting wrong results from it.