Liquidation of off-scale values in time series

user_10842659 0
Viewed 52

I have a problem like in the picture.

enter image description here

I figured out that I would do this with a moving average, but as you know the average leaves an unnecessary deviation and I need exact data, not average data. This is not an acceptable solution.

import bottleneck as bn

df['mT_Dok2'] = bn.move_mean(df['mT_Dok'], window=20, min_count=1)
df['mT_Dok'].plot(linewidth=0.9)
df['mT_Dok2'].plot(linewidth=0.9)

enter image description here

It doesn't work for me, I don't know what to do

df['mT_Dok2'] = df['mT_Dok'].apply(lambda x: x if x > (3*np.std(x)) else np.nan)

the necessary tools to eliminate such jumps as will appear in the time series. I am asking for help, colleagues ....

pandas time-series numpy python
1 Answers

The bottleneck function is smoothing the whole function and using standard deviation on the whole dataset also doesn't help because it's probably going to miss the outlier.

A solution that could work for your case, is to

  • identify outliers as values that differ more than a given threshold from the rolling mean
  • substitute these anomalous values with the rolling mean

Demo:

import numpy as np 
from matplotlib import pyplot as plt 
import pandas as pd

x = np.arange(1,100) 
y = 2 * x^2 + x  
y[30] = 320 # outlier 
plt.plot(x,y) 
plt.show()

graph with outlier

Using bottleneck:

import bottleneck as bn

df = pd.DataFrame({'x': x, 'y': y})
df['y2'] = bn.move_mean(df['y'], window=20, min_count=1)
df['y'].plot()
df['y2'].plot()
plt.show()

bottleneck - rolling mean

This removes just the outlier from the data (using 200 as the threshold)

df['rolling_median'] = df['y'].rolling(window=3).median()
df['y2'] = np.where((abs(df.y-df.rolling_median)>200), df.rolling_median, df.y)
df['y'].plot()
df['y2'].plot()
plt.show()

removed outlier from data

Check: in the dataframe column y2, only the value that diverges from the mean has been substituted

df.iloc[27:35]
# Out: 
#      x    y     y2  move_mean
# 27  28   38  38.00      38.90
# 28  29   37  37.00      39.50
# 29  30   28  28.00      39.70
# 30  31  320  54.35      54.35
# 31  32   98  98.00      58.15
# 32  33   97  97.00      61.95
# 33  34   96  96.00      66.15
# 34  35   99  99.00      70.35

Note that in general, finding outliers in time series (also known as "anomaly detection") is a tricky issue that can be solved in a variety of ways, for instance with LSTM prediction models if one wants to take into consideration periodicities in the series.

user_23147370
Related