I am trying to apply (LSTM + autoencoder) for anomaly detection on my sequential data having total of 64 days data , 3 months, and total 64 days, data is recorded for 24 hours a day. For clarity you can see my data overview in image given below. I have total 4 features
The problem i am facing is how can i mention sequence of data? i am following this tutorial https://towardsdatascience.com/lstm-autoencoder-for-anomaly-detection-e1f4f2ee7ccf they are shaping their data in sequence by this way
(total samples, time step, total number of features)
by giving timestep is equal to 1, how they used 1 ? and how should i choose timestep value. any one can help?
their code for shaping sequence data is this
X_train = X_train.reshape(X_train.shape[0], 1, X_train.shape[1])
print("Training data shape:", X_train.shape)
X_test = X_test.reshape(X_test.shape[0], 1, X_test.shape[1])
print("Test data shape:", X_test.shape)
The whole code is given below an my dat shape is
Training dataset shape: (1368, 12) Test dataset shape: (168, 12) mean training data has 1368 samples of 57 days ans test data has 168 samples of 7 days
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
df = pd.read_csv('416celldata.csv', parse_dates=True, index_col="timestamp"
)
train = df['2014-04-01 04:00:00+00:00': '2014-05-20 06:00:00+00:00' ]
test = df['2014-05-20 06:00:00+00:00':]
print("Training dataset shape:", train.shape)
print("Test dataset shape:", test.shape)
# normalize the data
scaler = MinMaxScaler()
X_train = scaler.fit_transform(train)
X_test = scaler.transform(test)
X_train = X_train.reshape(X_train.shape[0], 1, X_train.shape[1])
print("Training data shape:", X_train.shape)
X_test = X_test.reshape(X_test.shape[0], 1, X_test.shape[1])
print("Test data shape:", X_test.shape)