Initialization of the hidden states of torch.nn.lstm

user_10349608 0
Viewed 832

As the explanations of num_layers in this link: https://discuss.pytorch.org/t/what-is-num-layers-in-rnn-module/9843

if the output of hidden state of the first lstm is the input of the hidden state of the second lstm (number_layers=2 for torch.nn.lstm), why do we need to initialize the hidden state with the first dimension (representing the number of hidden states I suppose) being the num_layers? For example, the code below:

rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))

Thanks so much!

lstm pytorch
1 Answers

The answer in https://discuss.pytorch.org/t/what-is-num-layers-in-rnn-module/9843 in fact has not shown the hidden state for the second lstm layer, and only showed its input (which is the output of the hidden state of the first lstm layer).

Thus, for each input sample at a time step, the first lstm layer takes this sample as its input. The second lstm layer takes the output of the hidden state of the first lstm layer as its input, and it outputs the final answer corresponding to the input sample of this time step. At the same time, both lstm layers needs to initialize their hidden states.

Subsequently, the updated hidden states of the two lstm layers will be used as the initial hidden states of the same two lstm layers when the input is the sample of the next time step. The process continues for the samples of the following time steps, which is what we called recurrence.

user_103496080
Related