How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

user_13960084 0
Viewed 887

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like

df.fillna(method="")

which takes basically no effort.

If one or two iterations is enough to get useful results, which come very near the real missing value, then I am willing to take the effort to implement it. (Dataset length 100.000 up to 200mio rows)

If it needs to be optimized like a Neural Network itself which can be costly in terms of time, isnt it better to simply use an LSTM?

lstm pandas data-cleaning kalman-filter time-series
1 Answers

pykalman is a package for implementing a Kalman filter.

Kalman filters were first used to clean up data for guidance systems, taking noisy data and cleaning it up quickly for control systems.

Here is a simple example

from pykalman import KalmanFilter
import numpy as np

kf = KalmanFilter(transition_matrices = [[1, 1], [0, 1]], observation_matrices = [[0.1, 0.5], [-0.3, 0.0]])
measurements = np.asarray([[1,0], [0,0], [0,1]])  # 3 observations
kf = kf.em(measurements, n_iter=5)
(filtered_state_means, filtered_state_covariances) = kf.filter(measurements)
(smoothed_state_means, smoothed_state_covariances) = kf.smooth(measurements)

Discussions on how to set up the state space and the code are on https://pykalman.github.io and https://en.wikipedia.org/wiki/Kalman_filter

user_54783730
Related