Why are the number of parameters 10 in SimpleRNN?

Admin 1
Viewed 274

This was my python code:

 from tensorflow.keras.models import Sequential
 from tensorflow.keras.layers import SimpleRNN

 Model = Sequential([
      SimpleRNN(2, input_shape=(2,2))
 ])

 print(Model.summary())

The output was:

Model: "sequential_23"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
simple_rnn_10 (SimpleRNN)    (None, 2)                 10        
=================================================================
Total params: 10
Trainable params: 10
Non-trainable params: 0
_________________________________________________________________
None

I don't understand why the Number of parameters is 10.

recurrent-neural-network keras tensorflow machine-learning
1 Answers

A simple recurrent layer is defined by:

enter image description here

In your code, you have specified input_shape=(2,2) which indicates |x|=2. You also set |h|=2 and have no output layer definied. This leads to the following model:

enter image description here

Since everything is fully connected in the feed-forward input (black connections) and recurrent input (blue connections), you get 2x2 matrices (weight shape in orange). The bias is a single value that is fully connected to each h unit, i.e. there is one weight per unit (in total 2, see green connections).

So in summary:

W_xh is the input matrix: 2x2=4 parameters

W_hh is the recurrent matrix: 2x2=4 parameters

b_h is the bias: 2 parameters

So in total, you have 10 parameters.

user_8601960
Related