How to reproduce Keras SimpleRNN behaviour

user_1613983 0
Viewed 30

I'm trying to learn how keras.layers.SimpleRNN works by following a relatively straightforward tutorial (https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/). However, this tutorial assumes the input is scalar, and I've been unable to scale this to higher-dimensional inputs. Here is my attempt to reproduce the behaviour of a simple RNN with 1x2 input for 3 time steps:

import keras

inputs = np.array([[[1, 2]]])
inputs = np.repeat(inputs, repeats = 3, axis=1)
inputs.shape

model = keras.Sequential()
model.add(keras.Input(shape=(3, 2), name='input'))
model.add(layers.SimpleRNN(4, name='rnn', activation='linear'))
model.add(layers.Dense(5, name='output', activation='linear'))

outputs = model(inputs)

w_inputs = model.get_weights()[0]
w_hidden = model.get_weights()[1]
b_hidden = model.get_weights()[2]
w_dense = model.get_weights()[3]
b_dense = model.get_weights()[4]

h0 = np.zeros(4)
h1 = np.matmul(np.array([[1, 2]]), w_inputs) + h0 + b_hidden
h2 = np.matmul(np.array([[1, 2]]), w_inputs) + b_hidden
h3 = np.matmul(np.array([[1, 2]]), w_inputs) + b_hidden
o3 = np.matmul(h3, w_dense) + b_dense

print(f'output: {outputs}')
print(f'expected: {o3}')

The output generated from keras and the values I calculated myself don't match:

output: [[-6.937807    4.0890574  -0.15574443 -1.2262737   1.9948364 ]]
expected: [[-2.38697288  1.03174649 -1.44180991 -1.28951854  0.39630487]]

why?

keras
1 Answers

The matrix multiplications are correct, but they need to be performed at every timestep. And if return_sequences=False Keras returns the output of the last timestep of shape ( 1 , 4 ). So, we can create a for-loop which can perform the matrix multiplications for 3 timesteps.

Note: The input shape ( 3 , 2 ) denotes ( time_steps , num_features )

Here's the code to replicate the Keras RNN model,

from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

model = keras.Sequential()
model.add(keras.Input(shape=(3, 2), name='input'))
model.add(layers.SimpleRNN(4, name='rnn', activation='linear' ))
model.add(layers.Dense(5, name='output', activation='linear'))

X = [ [ [ 2. , 3. ] , [ 4. , 5. ] , [ 0.1 , 0.2 ] ] ]
X = np.array( X )
print( 'model output >> {}'.format( model( X ) ) )

weights = model.get_weights()
U = weights[ 0 ]
H = weights[ 1 ]
b_U = weights[ 2 ]
O = weights[ 3 ]
b_O = weights[ 4 ]
h_0 = np.zeros( ( 1 , 4 ) )

sample = X[ 0 ]
for t in range( sample.shape[ 0 ] ):
    # Transform the shape from ( 2 , ) to ( 1 , 2 ) for matmul
    features = np.expand_dims( sample[ t ] , axis=0 )
    # linear activation a( x ) = x
    cell_out = np.matmul( h_0 , H ) + np.matmul( features , U ) + b_U
    # The output of this timestep becomes the hidden state for the next timestep
    h_0 = cell_out

# Affine/Dense layer operation
output = np.matmul( cell_out , O ) + b_O
print( 'numpy output {}'.format( output ) )

The output of the code,

model output >> [[-0.43817645  0.6968174  -3.286867    3.1212366   0.24934107]]
numpy output [[-0.43817637  0.69681754 -3.28686709  3.12123692  0.24934101]]
user_135464260
Related