How to handle variable length sentences in PyTorch with Glove embedding layer?

user_9040134 0
Viewed 332

I am building a text classifier using an RNN in PyTorch. The embeddings i'm using are GLOVE. However i am feeding variable length index references in to the model. This will lead to variable length embeddings, which i take it will not work. How do i get around this and make the embedding output the same length for all sentences?

def forward(self, sentence):
        embeds = self.embedding(sentence)
        hidden = self.__init__hidden(size) 
        output, hidden = self.rnn(embeds, hidden)
        out = self.hidden2out(output)

Also, if someone could tell me how to choose the hidden layer size that would be great.

pytorch deep-learning machine-learning nlp python
1 Answers

Your data in this case can be represented in pytorch as having shape: [seq_len, batch, vocab_size]. When you create a tensor for your model this way, the batch size is up to you, and sequence length is variable.

When you employ an embedding, such as GloVe or word2vec, what you're doing is condensing your vocab size (from many thousands to <1000) to make it more information dense and to give your model some contextual information about each word. So you can see that your output from an embedding layer will be of shape: [seq_len, batch, embedding_size]. Therefore, it doesn't matter how long your sequences are at this stage because your output is sequence-length-independent.

Without knowing your embedding layer, I just have to assume that it's operating along the correct dimensions. If it is not, a couple of transpose operations to ensure that it is will work.

The only way to gain intuition about hidden layer size (I'm assuming you're not talking about your h_0 input into your RNN) and number of hidden layers is to unfortunately try things about. I can't really even give you a guess because I have no idea what dataset you're using or what problem you're trying to solve. There is no one-size-fits-all policy in machine learning!

user_104757620
Related