Which axis does Keras SimpleRNN / LSTM use as the temporal axis by default?

user_1422096 0
Viewed 307

When using a SimpleRNN or LSTM for classical sentiment analysis algorithms (applied here to sentences of length <= 250 words/tokens):

model = Sequential()
model.add(Embedding(5000, 32, input_length=250))   # Output shape: (None, 250, 32)
model.add(SimpleRNN(100))                          # Output shape: (None, 100)
model.add(Dense(1, activation='sigmoid'))          # Output shape: (None, 1)

where is it specified which axis of the input of the RNN is used as the "temporal" axis?

To be more precise, after the Embedding layer, a given input sentence, e.g. "the cat sat on the mat", is encoded into a matrix x of shape (250, 32), where 250 is the max length (in words) of the input text, and 32 the dimension of the embedding. Then, where in Keras is it specified if this will be used:

  1. h[t] = activation( W_h * x[:, t] + U_h * h[t-1] + b_h )

or this:

  1. h[t] = activation( W_h * x[t, :] + U_h * h[t-1] + b_h )

(In both cases, y[t] = activation( W_y * h[t] + b_y ))

TL;DR: if an input for a RNN Keras layer is of size, say, (250, 32), which axis does it use as the temporal axis by default? Where is this detailed in the Keras or Tensorflow documentation?

PS: how to explain the number of parameters (given by model.summary()) which is 13300? W_h has 100x32 coefs, U_h has 100x100 coefs, b_h has 100x1 coefs, i.e. we already have 13300! There is no coefs left for W_y and b_y! How to explain this?

recurrent-neural-network keras tensorflow python
1 Answers

Temporal axis: it's always dim 1, unless time_major=True, then it's dim 2; the Embedding layer outputs a 3D tensor. This can be seen here where step_input_shape is the shape of input fed to the RNN cell at each step in the recurrent loop. For your case, timesteps=250, and the SimpleRNN cell "sees" a tensor shaped (batch_size, 32) at each step.

# of params: you can see how the figure's derived by inspecting each layer's .build() code: Embedding, SimpleRNN, Dense, or likewise calling .weights on each layer. For your case, w/ l = model.layers[1]:

  • l.weights[0].shape == (32, 100) --> 3200 params (kernel)
  • l.weights[1].shape == (100, 100) --> 10000 params (recurrent_kernel)
  • l.weights[2].shape == (100,) --> 100 params (bias) (sum: 13,300)

Computation logic: there is no W_y or b_y; the "y" is the hidden state, h, actually for all recurrent layers - what you cite are likely from generic RNN formulae. @ "in both cases..." - this is false; to see what's actually happening, inspect the .call() code.

P.S. I recommend defining the full batch_shape of the model for debugging, as it eliminates the ambiguous None shapes

SimpleRNN formula vs. code: as requested; note the h in source code is misleading, and is typically z in formulae ("pre-activation").

enter image description here

  • return_sequences=True -> outputs for all timesteps are returned: (batch_size, timesteps, channels)

  • return_sequences=False -> only last timestep's output is returned: (batch_size, 1, channels). See here

user_101337970
Related