I am a beginner in the Deep Learning field and I need help with LSTMs.
The data I analyze consists of a number of http errors for each timestep for a period of 1 month. I chose to consider data for every 10min timestep (in order to have more data for the training).
Head of the data :
The data shows seasonality trends, there is a very low number of errors during the week ends and a great number during the working days (ie when people connect to the servers).
My goal is to predict the number of errors for the next day. I created a new column with the data shifted by a day in order to predict it (shifted by 6*24 considering that we have 10min timesteps). So I'm trying to predict 6*24 timesteps ahead.
My model is the following :
model = Sequential()
model.add(LSTM(1008, return_sequences = True, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dropout(0.2))
model.add(LSTM(144, return_sequences = True, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dropout(0.2))
model.add(LSTM(7, return_sequences = True, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
I learnt that, in order to train the model, the data must have the following shape : [samples, timesteps, features]. It consists of several sub time series. So first I tried the following shape : [60, 6*24, 1], so that is 60 samples and each one represents a day.
I also tried several number of layers and hidden units but my problem is always the same : the model can't capture the weekly seasonality of the data, it seems like the predictions are shifted or maybe it makes predictions based on the previous day.
Here is the plot I have for the forecasting:
Test data consists of 12 days following the month of the training data.
I thought that if I specify to the model that the sub time series are weeks it will resolve the problem. So I also tried to gather more data and to consider the following shape for the input : [21, 6*24*7, 1], so that is 21 samples and each one represents a week (10min*6*24*7). It didn't work and I have a plot very similar to the previous one.
I hope that my explanations are clear, if not feel free to ask for more details.
Thanks
PS: If the only way is to consider 1Hour or one day timesteps when gathering the data let me know. I didn't investigate this option because I would have very few number of training examples.