Can someone explain to me the difference between activation and recurrent activation arguments passed in initialising keras lstm layer?

Viewed 9878

Can someone explain to me the difference between activation and recurrent activation arguments passed in initialising keras lstm layer?

According to my understanding LSTM has 4 layers. Please explain what are th e default activation functions of each layer if I do not pass any activation argument to the LSTM constructor?

5 Answers

On code

Line from 1932

i = self.recurrent_activation(z0)
f = self.recurrent_activation(z1)
c = f * c_tm1 + i * self.activation(z2)
o = self.recurrent_activation(z3)
h = o * self.activation(c)

recurrent_activation is for activate input/forget/output gate.

activation if for cell state and hidden state.

According to the explanation by Andrew Ng in this video1 , the three gates namely update,forget and output gate require a sigmoid-type activation function.Hence the activation in keras documentation refers to these acativation values.

The activations required for the update candidate and the output are tanh. So the recurrent activations correspond to these activations in the Keras documentation.RA-Recurrent Activation, Act.-Activations

I verified your question and below is my conclusion: - activation:tanh; - recurrent activation:sigmoid(default hard_sigmoid in keras); and for 4 gate in one cell: - i、f、o use recurrent activation; - C and update h use activation;

Related