How to debug RNNs?

user_12228692 0
Viewed 54

I have set up a ResNet50 network for an optical application. With two input images, the network gives an estimate of 65 values (regression) and it works pretty well. However, the two input images belong to a time series, and the images of the time series will be somewhat correlated over a span of 10-15 times, so I expect that an additional RNN could improve estimates. I have tried to set up the network shown in the figure, using mostly frozen ResNet50 parameter values found by separate training and “TimeDistributed” ResNet50s. However the RNN training does not give useful accuracy.

Full LSTM network

I have now spent 2-3 weeks trying to debug my code (in particular the generator) but I have not found any coding errors. In frustration, I tried to set up the simplest RNN I could think of: A complete Resnet50 with either one or two SimpleRNNs with linear activation. However they do not provide even nearly the same accuracy as the ResNet50 alone in spite of the correlated time series.

SimpleRNN network

So my question is: Is it correct to assume that a single SimpleRNN with linear activation should provide the same accuracy as the ResNet50 alone?

recurrent-neural-network resnet lstm keras regression
1 Answers

This is a bit speculative, but it might suggest an approach to debug the RNN and answer your question. Here is an extremely simple network with a SimpleRNN and a test input of 2 samples, each with a single time step and single feature: i.e. shape=(2,1,1)

from keras.models import Sequential
from keras.layers import SimpleRNN
import numpy as np

x_train=np.array([[[0.1]],
                  [[0.2]]])

y_train=np.array([[1],[0]])
print(x_train.shape)
print(x_train)

print(y_train.shape)
print(y_train)

#simple network
model = Sequential()
model.add(SimpleRNN(1,activation=None, use_bias=False, input_shape=(1,1)))
model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])

model.fit(x_train, y_train, epochs=10, batch_size=2)

wgt=model.get_weights()
print(wgt)
print('model.predict(x_train)')
print(model.predict(x_train))

Based on running the above, two weights come out of the RNN network. The first seems to be a simple scaling of the input and the second I'm suspecting is the weight of the recurrent loop which is not actually used for a single time step as in this example. The activation is linear so the result then matches the model.predict.

You may be able to extend approach this to reason about the performance with the Resnet and potentially answer your question. I hope this helps.

user_70240060
Related