I am building a sequence-to-label classifier, where the input data are text sequences and output labels are binary. The model is very simple, with GRU hidden layers and a Word Embeddings input layer. I want a [n, 60] input to output a [n, 1] label, but the Torch model returns a [n, 60] output.
My model, with minimal layers:
class Model(nn.Module):
def __init__(self, weights_matrix, hidden_size, num_layers):
super(Model, self).__init__()
self.embedding, num_embeddings, embedding_dim = create_emb_layer(weights_matrix, True)
self.hidden_size = hidden_size
self.num_layers = num_layers
self.gru = nn.GRU(embedding_dim, hidden_size, num_layers, batch_first=True)
self.out = nn.Linear(hidden_size, 1)
def forward(self, inp, hidden):
emb = self.embedding(inp);
out, hidden = self.gru(emb, hidden)
out = self.out(out);
return out, hidden;
def init_hidden(self, batch_size):
return torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device);
Model Layers:
Model(
(embedding): Embedding(184901, 100)
(gru): GRU(100, 60, num_layers=3, batch_first=True)
(out): Linear(in_features=60, out_features=1, bias=True)
)
Input shapes of my data are: X : torch.Size([64, 60]), and Y : torch.Size([64, 1]), for a single batch of size 64.
When I run the X tensor through the model, it should output a single label, however, the output from the classifier is torch.Size([64, 60, 1]). To run the model, I do the following:
for epoch in range(1):
running_loss = 0.0;
batch_size = 64;
hidden = model.init_hidden(batch_size)
for ite, data in enumerate(train_loader, 0):
x, y = data[:,:-1], data[:,-1].reshape(-1,1)
optimizer.zero_grad();
outputs, hidden = model(x, hidden);
hidden = Variable(hidden.data).to(device);
loss = criterion(outputs, y);
loss.backward();
optimizer.step();
running_loss = running_loss + loss.item();
if ite % 2000 == 1999:
print('[%d %5d] loss: %.3f'%(epoch+1, ite+1, running_loss / 2000))
running_loss = 0.0;
When I print the shape of outputs, it is 64x60x1 rather than 64x1. What I also don't get is how the criterion function is able to calculate the loss when the shapes of outputs and labels are inconsistent. With Tensorflow, this would always throw an error, but it doesn't with Torch.