How to initialize decoder's state when it has multiple layers?

Viewed 411

I'm using PyTorch/Tensorflow to implement a seq2seq model.

When the encoder has m layers while the decoder has n layers, should one use only the state of the last encoder layer to initialize the state of the first decoder layer and keep all the states of layers above zeros?

Or, when m == n, should one use all states of the encoder to initialize the states of the corresponding layer of the decoder?

If the encoder(decoder) is bidirectional while the decoder(encoder) is not, how to initialize the hidden state of the decoder in this situation?

0 Answers
Related