I am passing a batch of sentences to the GRU. The sentences are of varying length and have been padded (padding_idx = 0) to form the batch. The sentences vary a lot in lenghts, from 32 words to 105 words. I have specified the padding_idx while creating the embedding layer. If I am not using nn.pack_padded_sequence, is any kind of masking information is still required or will it be taken care of internally?
Also, after obtaining the hidden_states from the GRU, I want to form sentence representations. For this, I am applying a soft-attention mechanism on the hidden_states obtained for a sentence. Basically, I obtain a score for each hidden state and take the weighted mean. This would also assign a weight to a hidden state corresponding padding_idx. Is it correct or am I required to use masking here? Like setting the weight for the hidden state corresponding to the padding_idx to a very small value.