Why is my Tensorflow LSTM Timeseries model returning only one value into the future

Question

I have a tensorflow model for predicting Timeseries values using LSTM, it trains fine but when I ask it to predict some values in time it only gives me the T+1 value,

How can I make it to give me from T+1 to T+n values instead of just T+1.

I thought about giving him back the predicted value to analyse again in a loop, e.g.

We look back at 20 samples for this example

T±0 = now value
T-k = value k steps into the past (known)
T+n = value n steps into the future (unkown at the start)

--- Algorithm

T+1 = model.predict(data from T-20 to T±0)
T+2 = model.predict(data from T-19 to T+1) #using the previously found T+1 value
T+3 = model.predict(data from T-18 to T+2) #using the previously found T+1 and T+2 values
.
.
.
T+n = model.predict(data from T-(k-n) to T+(n-1)) # using the previously found T+1 .. T+(n-1) values

The thing is that T+1 has an mean absolute error of around 0.75%, doesn't the error propagate/compound through the predictions ? If it does it means that if I ask the program to predict T+10 it will have a mean absolute error of 0.75%^10 = ~7.7%, which is not very good in my case. So I'm looking for other ways to predict up to T+n values.

I've looked at few youtube tutorials but each time it seems that their call of model.predict(X) returns multiple values already, and I have no idea about what parameters I could have missed.

Code :

import tensorflow.keras as tfks
import pandas as pd
import numpy as np

def model_training(dataframe,folder_name, window_size=40, epochs=100, batch_size=64):
    """Function to start training the model on given data
    
    Parameters
    ----------
    dataframe : `pandas.DataFrame` The dataframe to train the model on
    window_size : `int` The size of the lookback window to use
    epochs : `int` The number of epochs to train for
    batch_size : `int` The batch size to use for training

    Returns
    -------
    None
    """
    dataframe,_ = Process.pre(dataframe) #function to standardize each column of the data

    TRAIN_SIZE = 0.7
    VAL_SIZE = 0.2
    TEST_SIZE = 0.1

    #Splitting the data into train, validation and test sets
    x,y = dataframe_to_xy(dataframe, window_size) #converts pandas dataframe to numpy array

    x_train,y_train = x[:int(len(dataframe)*TRAIN_SIZE)],y[:int(len(dataframe)*TRAIN_SIZE)]
    x_val,y_val = x[int(len(dataframe)*TRAIN_SIZE):int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE))],y[int(len(dataframe)*TRAIN_SIZE):int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE))]
    x_test,y_test = x[int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE)):],y[int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE)):]

    #Creating the model base
    model = tfkr.models.Sequential()
    model.add(tfkr.layers.InputLayer(input_shape=(window_size, 10)))
    model.add(tfkr.layers.LSTM(64))
    model.add(tfkr.layers.Dense(8, 'relu'))
    model.add(tfkr.layers.Dense(10, 'linear'))

    model.summary()

    #Compiling and saving the model
    cp = tfkr.callbacks.ModelCheckpoint('ai\models\'+folder_name+'\', save_best_only=True)
    model.compile(loss=tfkr.losses.MeanSquaredError(), optimizer=tfkr.optimizers.Adam(learning_rate=0.0001), metrics=[tfkr.metrics.RootMeanSquaredError()])

    model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=(x_val, y_val), callbacks=[cp])

def predict_data(model,data_pre,window_size,n_future):
    '''Function to predict data using the model
    
    Parameters
    ----------
    model : `tensorflow.keras.models.Sequential` The model to use for prediction
    data_pre : `pandas.DataFrame` The dataframe to predict on
    window_size : `int` The size of the lookback window to use
    n_future : `int` Number of values to predict in the future

    Returns
    -------
    data_pred : `pandas.DataFrame` The dataframe containing the predicted values
    '''

    time_interval = data_pre.index[1] - data_pre.index[0] 

    #Setting up the dataframe to predict on
    data_pre, proc_params = Process.pre(data_pre) #function to standardize each column of the data
    data_pred = data_pre.iloc[-window_size:]
    data_pred = data_pred.to_numpy().astype('float32')
    data_pred = data_pred.reshape(1,window_size,10)

    #Predicting the data
    data_pred = model.predict(data_pred)
    
    #Converting the data from numpy array to pandas dataframe and doing some formatting + post-processing/reversing standardization
    #yada yada pandas dataframe

    return data_pred

If there is no way preventing the error propagation, do you have any tips for reducting the error of the model ?

Thanks in advance.

Gautam Chettiar · Answer

One way which you can do this is, lets assume you initially trained your model something like this:

x_train = [
[1, 2, 3, 4, 5, 6, 7],
[2, 3, 4, 5, 6, 7, 8],
]
y_train = [
[8],
[9]
]

And the training was done as such using the model.fit with all appropriate reshaping.

model.fit(x_train, y_train)

Then, you would expect something like this when you try to predict

ans = model.predict(x_train[0])
ans: 8

But lets say you wanted it predict the next value as well, with the expected output as 8 and then 9, or for n expected outputs, you would expect a result such as 8, 9, 10, 11... and so on.

Then I suggest to approach the problem the following way:

def predict_continuous(model, input_arr, num_ahead):
    # Your input arr was [1,2,3,4,5,6,7]
    # You want the output to go beyond just 8, lets say 8,9,10,11,...
    # num_ahead: How much into the future you need predictions.
    predictions = []
    model_input = input_arr
    for i in range(num_ahead):
        predictions.append(model.predict(model_input)) # Get the prediction, and subsequent predictions are appended
        # Since the LSTM model is trained for a particular input size, we need to keep dynamically changing the input
        # so we strip off the first value from the input array, and append the latest prediction as its last value.
        model_input = model_input[1:].append(predictions[-1])
    return predictions

Now this won't work directly, you will have to do some reshaping, however the core idea remains the same.

AndrzejO · Answer

You could let your LSTM return the full sequences, not only the last output, like this:

model = tfkr.models.Sequential()
    model.add(tfkr.layers.InputLayer(input_shape=(window_size, 10)))
    model.add(tfkr.layers.LSTM(64, return_sequences=True))
    model.add(tfkr.layers.Dense(8, 'relu'))
    model.add(tfkr.layers.Dense(10, 'linear'))

Each LSTM output would go through the same dense weights (because we did not flatten). Your output would be then of shape

(None, window_size, 10)

This means that for k input time points you would get k output time points.

Downside is, that the first output would be calculated only using the first input. The second output only by using the first two inputs, and so on. So I would suggest to use a bidirectional LSTM, and combine both directions, maybe like this:

model = tfkr.models.Sequential()
        model.add(tfkr.layers.InputLayer(input_shape=(window_size, 10)))
        model.add(tfkr.layers.Bidirectional(tfkr.layers.LSTM(64, return_sequences=True), merge_mode='sum')
        model.add(tfkr.layers.Dense(8, 'relu'))
        model.add(tfkr.layers.Dense(10, 'linear'))

Why is my Tensorflow LSTM Timeseries model returning only one value into the future

2 Answers