How to make an accuracy comparison for time series prediction using LSTM and RNN model?

Question

Hi I am currently developing a project for bus arrival time prediction system using LSTM and RNN. These two framework will be compared for their results of predicting the delay_in_min data that I have included in the dataset. I need help with some problems that I have encountered while developing this project.

I have try to run my dataset with this framework for both LSTM and RNN.

**This is a part of my dataset:

For data preprocessing

X.shape = (315, 5, 21)
y.shape = (315,)
I split the data into three part(train:200, validation:50 and test:65).

LSTM:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import *
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.losses import MeanSquaredError
from tensorflow.keras.metrics import RootMeanSquaredError
from tensorflow.keras.optimizers import Adam

model1 = Sequential()
model1.add(InputLayer((5,21)))
model1.add(LSTM(256, activation='relu', return_sequences=True))
model1.add(LSTM(256, activation='relu'))
model1.add(Dense(1))
model1.summary()

cp = ModelCheckpoint('model1/', save_best_only=True)
model1.compile(loss=MeanSquaredError(), optimizer=Adam(learning_rate=0.0001), metrics=[RootMeanSquaredError()])
model1.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=1000, callbacks=[cp])

RNN:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import *
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.losses import MeanSquaredError
from tensorflow.keras.metrics import RootMeanSquaredError
from tensorflow.keras.optimizers import Adam

model1 = Sequential()
model1.add(InputLayer((5,21)))
model1.add(SimpleRNN(256, activation='relu', return_sequences=True))
model1.add(SimpleRNN(256, activation='relu'))
model1.add(Dense(1))
model1.summary()

cp = ModelCheckpoint('model1/', save_best_only=True)
model1.compile(loss=MeanSquaredError(), optimizer=Adam(learning_rate=0.0001), metrics=[RootMeanSquaredError()])
model1.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=1000, callbacks=[cp])

However, the RMSE metrics could only achieved the lowest value at 2.4580 and there are no changes happens at some point for 1000 epochs. When I look to the results of prediction, some of the value is too far from the expected output value which was supposed to be between this three values(0,5 and 10). I have tried to change the parameter value by increasing or decreasing it but somehow I don't know if I can do that or not. To make it more clear, below is the example of the output:

Epoch 1/1000
6/7 [========================>.....] - ETA: 0s - loss: 8263.2939 - root_mean_squared_error: 90.9027INFO:tensorflow:Assets written to: model1/assets
INFO:tensorflow:Assets written to: model1/assets
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
7/7 [==============================] - 7s 758ms/step - loss: 7940.8867 - root_mean_squared_error: 89.1117 - val_loss: 331.7596 - val_root_mean_squared_error: 18.2143
Epoch 2/1000
6/7 [========================>.....] - ETA: 0s - loss: 404.0868 - root_mean_squared_error: 20.1019INFO:tensorflow:Assets written to: model1/assets
INFO:tensorflow:Assets written to: model1/assets
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
7/7 [==============================] - 4s 583ms/step - loss: 390.3654 - root_mean_squared_error: 19.7577 - val_loss: 34.9191 - val_root_mean_squared_error: 5.9092
Epoch 3/1000
7/7 [==============================] - ETA: 0s - loss: 33.4639 - root_mean_squared_error: 5.7848INFO:tensorflow:Assets written to: model1/assets
INFO:tensorflow:Assets written to: model1/assets
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
7/7 [==============================] - 4s 637ms/step - loss: 33.4639 - root_mean_squared_error: 5.7848 - val_loss: 11.8592 - val_root_mean_squared_error: 3.4437
Epoch 4/1000
6/7 [========================>.....] - ETA: 0s - loss: 20.1906 - root_mean_squared_error: 4.4934INFO:tensorflow:Assets written to: model1/assets
INFO:tensorflow:Assets written to: model1/assets
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
7/7 [==============================] - 4s 581ms/step - loss: 20.0047 - root_mean_squared_error: 4.4727 - val_loss: 8.8859 - val_root_mean_squared_error: 2.9809
Epoch 5/1000
7/7 [==============================] - ETA: 0s - loss: 14.3730 - root_mean_squared_error: 3.7912INFO:tensorflow:Assets written to: model1/assets
INFO:tensorflow:Assets written to: model1/assets
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
7/7 [==============================] - 3s 552ms/step - loss: 14.3730 - root_mean_squared_error: 3.7912 - val_loss: 7.1798 - val_root_mean_squared_error: 2.6795
Epoch 6/1000
6/7 [========================>.....] - ETA: 0s - loss: 13.1876 - root_mean_squared_error: 3.6315INFO:tensorflow:Assets written to: model1/assets
INFO:tensorflow:Assets written to: model1/assets
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming  to avoid naming conflicts when loading with `tf.keras.models.load_model`. If renaming is not possible, pass the object in the `custom_objects` parameter of the load function.
7/7 [==============================] - 4s 709ms/step - loss: 12.8867 - root_mean_squared_error: 3.5898 - val_loss: 5.9226 - val_root_mean_squared_error: 2.4336
Epoch 7/1000
7/7 [==============================] - 0s 64ms/step - loss: 12.3348 - root_mean_squared_error: 3.5121 - val_loss: 6.0180 - val_root_mean_squared_error: 2.4532
Epoch 8/1000
7/7 [==============================] - 0s 59ms/step - loss: 12.2133 - root_mean_squared_error: 3.4948 - val_loss: 7.3143 - val_root_mean_squared_error: 2.7045
Epoch 9/1000
7/7 [==============================] - 0s 52ms/step - loss: 12.1458 - root_mean_squared_error: 3.4851 - val_loss: 5.9605 - val_root_mean_squared_error: 2.4414
Epoch 10/1000
.
.
.
Epoch 991/1000
7/7 [==============================] - 0s 61ms/step - loss: 6.4219 - root_mean_squared_error: 2.5341 - val_loss: 3.4269 - val_root_mean_squared_error: 1.8512
Epoch 992/1000
7/7 [==============================] - 0s 60ms/step - loss: 5.7453 - root_mean_squared_error: 2.3969 - val_loss: 12.1370 - val_root_mean_squared_error: 3.4838
Epoch 993/1000
7/7 [==============================] - 0s 58ms/step - loss: 7.3273 - root_mean_squared_error: 2.7069 - val_loss: 4.5648 - val_root_mean_squared_error: 2.1365
Epoch 994/1000
7/7 [==============================] - 0s 64ms/step - loss: 5.7639 - root_mean_squared_error: 2.4008 - val_loss: 3.5638 - val_root_mean_squared_error: 1.8878
Epoch 995/1000
7/7 [==============================] - 0s 67ms/step - loss: 5.2103 - root_mean_squared_error: 2.2826 - val_loss: 6.0134 - val_root_mean_squared_error: 2.4522
Epoch 996/1000
7/7 [==============================] - 0s 66ms/step - loss: 5.5951 - root_mean_squared_error: 2.3654 - val_loss: 3.3826 - val_root_mean_squared_error: 1.8392
Epoch 997/1000
7/7 [==============================] - 0s 65ms/step - loss: 5.4926 - root_mean_squared_error: 2.3436 - val_loss: 4.6553 - val_root_mean_squared_error: 2.1576
Epoch 998/1000
7/7 [==============================] - 0s 68ms/step - loss: 5.6556 - root_mean_squared_error: 2.3781 - val_loss: 3.2487 - val_root_mean_squared_error: 1.8024
Epoch 999/1000
7/7 [==============================] - 0s 68ms/step - loss: 5.5924 - root_mean_squared_error: 2.3648 - val_loss: 3.1579 - val_root_mean_squared_error: 1.7770
Epoch 1000/1000
7/7 [==============================] - 0s 66ms/step - loss: 6.3305 - root_mean_squared_error: 2.5161 - val_loss: 6.0776 - val_root_mean_squared_error: 2.4653

Next, is I don't know how to retrieve the latest RMSE value that have been saved to the '/model1' file. From my understanding, this coding will save the best RMSE value to the that file after compare it with the value at the previous checkpoint. I need this value to compare the accuracy of the prediction made by both LSTM and RNN. Is there any way for me to retrieve it?

user_6299772 · Answer

For the first part of your question: If your model "can't quite get the underlying problem" this normally means, that your model is too simple. But this can also have a lot of reasons:

Your trying to predict arrival times, maybe they are completely random?
Maybe they depend on parameters you don't feed into the model?
Maybe your data preprocessing is bad? You could try to improve your question by posting how your data is looking and how your preprocessing is done. Also you could try to add a few more Dense layers at the end to increase your models complexity.

For the second part of your question: You can add multiple callbacks to your training. Currently you are only adding the callback, which will save the best model. In addition to that you can look here how to create a custom callback, which can, for example, save the best loss value to a .txt file, so that you can retrieve it later.

How to make an accuracy comparison for time series prediction using LSTM and RNN model?

1 Answers