I'm training a LSTM variant, PhasedLSTM, for regression. I'm using tensorflow.contrib.rnn.PhasedLSTMCell which expects a vector of timestamps in addition to the features. Here's my model definition :
from tensorflow.keras.layers import Dense, RNN, Input
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.contrib.rnn import PhasedLSTMCell
hidden_size = 128
num_features = 217
num_timesteps = 2016
input_layer_timestamps = Input(shape=(num_timesteps, 1), name='time_input')
input_layer_features = Input(shape=(num_timesteps, num_features) name='data_input')
timed_inputs = (input_layer_timestamps, input_layer_features)
P_cell = PhasedLSTMCell(hidden_size)
PLSTM_layer = RNN(P_cell, return_sequences=False, name='Phased_LSTM_1')(timed_inputs)
output_layer = Dense(2, activation=None)(PLSTM_layer_1)
model = Model(inputs = [input_layer_timestamps, input_layer_features],
outputs = [output_layer])
lstm_optimizer = Adam(lr=Adam_lr, clipnorm=5.)
model.compile(optimizer=lstm_optimizer,
loss='mse')
model.summary()
The model compiles and trains nicely. Validation results seem good, with reasonable error. However the output of model.summary() from the previous snippet is :
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
time_input (InputLayer) (None, 2016, 1) 0
__________________________________________________________________________________________________
data_input (InputLayer) (None, 2016, 217) 0
__________________________________________________________________________________________________
Phased_LSTM_1 (RNN) (None, 128) 0 time_input[0][0]
data_input[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 2) 66 Phased_LSTM_1[0][0]
==================================================================================================
Total params: 258 Trainable params: 258 Non-trainable params: 0
__________________________________________________________________________________________________
Notably with the number of trainable parameters 0 for Phased_LSTM_1
If model.weights() is then called, it does not show the LSTM part at all :
ipdb> NN.weights
[<tf.Variable 'scope/dense/kernel:0' shape=(128, 2) dtype=float32>, <tf.Variable 'scope/dense/bias:0' shape=(2,) dtype=float32>]
Additionally if the model is then saved with model.save() and loaded with either tensorflow.keras.models.load_model or defining model -> model.load() the model will behave completely differently. The reasonably good validation accuracy will change into a model that predicts only noise, i.e. it looks like the original fairly good regression model was changed from a RNN into something that indeed looks like a 128-neuron single layer NN with linear activation trained on the wrong thing.
I have three questions:
1) What's going on? Is this a bug or just me abusing the Keras RNN API?
2) How can I verify that the LSTM is being trained properly? I want to inspect the layers like I would normally with model.weights and model.get_weights()
3) How can I save and load this model without breaking it? My input and output pipelines and the module around it are built for Keras, so I'd rather not convert the NN definition to 'base' tensorflow if possible.
I'm using tensorflow-gpu 1.13.1 with Keras 2.2.4-tf