I'm writing my first neural machine translator in tensorflow. I am using an encoder/decoder mechanism with attention. My encoder and decoder are lstm stacks with residual connections, but the encoder has an initial bidirectional layer. The decoder does not.
It is common practice in the code that I have seen to initialize the state of the decoder cells with the last state of the encoder cells. However, this is only a clean solution if your encoder and decoder architectures are the same, as is the case in many of the seq2seq tutorials. In many other systems, such as this model by Google, the architectures differ on the encoder and decoder.
What are some of the alternative strategies used for initializing decoder state in these circumstances?
I have seen cases where the encoder's last hidden state is passed through a trained weight vector to create the initial decoder state, for all decoder layers. I have also seen more inventive ideas such as the one presented here, but I would like to develop an intuition as to why people pick certain strategies.