I'm building a LSTM model with variable length arrays as input. In many resources, I was recommended to do padding which is inserting 0 until all input arrays have the same length and then applying Masking for the model to ignore the 0s.
However, after many trainings, I feel like Masking does not work as expected, the padded 0s in the input still affect the prediction ability of the model.
After concatenating all sequences into one array, my training data looks like below without padding:
X y
[1 2 3] 4
[2 3 4] 5
[3 4 5] 6
...
My python implementation:
import numpy as np
import pandas as pd
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Masking
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator
""" Raw Training Input """
arr = np.array([
[1, 2, 3, 4, 5, 6],
[5, 6, 7],
[11, 12, 13, 14]
], dtype=object)
timesteps = 3
n_features = 1
maxlen = 6
""" Padding """
padded_arr = pad_sequences(arr, maxlen=maxlen, padding='pre', truncating='pre')
""" Concatenate all sequences into one array """
sequence = np.concatenate(padded_arr)
sequence = sequence.reshape((len(sequence), n_features))
# print(sequence)
""" Training Data Generator """
generator = TimeseriesGenerator(sequence, sequence, length=timesteps, batch_size=1)
""" Check Generator """
for i in range(len(generator)):
x, y = generator[i]
print('%s => %s' % (x, y))
""" Build Model """
model = Sequential()
model.add(Masking(mask_value=0.0, input_shape=(timesteps, n_features))) # masking to ignore padded 0
model.add(LSTM(1024, activation='relu', input_shape=(timesteps, n_features), return_sequences=False))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(generator, steps_per_epoch=1, epochs=1000, verbose=10)
""" Prediction """
x_input = np.array([2,3,4]).reshape((1, timesteps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat) # here I'm expecting 5 because the input is [2, 3, 4]
For the prediction, I input [2,3,4] and most of the time I keep getting values very far away from the expected value (= 5)
I'm wondering if I missed out on something or simply because the model architecture was not correctly tuned.
I want to understand why the model is not predicting correctly. Is the Masking the issue or is it something else?