After checking the official doc here keras mask tutorial, it is still not clear to me whether Keras Dense layer can propagate the mask to its following layers 4 and 5 in below example.
Another question is, when calculating the loss at the 5th layer, shall we apply the mask?
We can say it is not needed because the 2nd layer LSTM already ignored those <pad> tokens in input sequences. However, I've read somewhere that the LSTM output of <pad> tokens are NOT zero, but following the last valid token's output. It will affect the value of loss. Thus, we need to apply the mask at the 5th layer?
Our input are padded sequences, and we have a sequential model in Keras
- Embedding layer with mask_zero = True //can generate mask
- LSTM layer //can consume mask
- Dense layer //Question: can this layer propagate mask to other layers in this model
- other layers...
- output layer with
sigmoidas activation function
Thanks for your kind help!