Classification in LSTM returns same value for classification

user_16413631 0
Viewed 82

This is my first time posting in stack overflow so forgive me if I do any sort of mistake.

I have 10000 data, and each data has a label of 0 and 1. I want to perform classification using LSTM as this is time series data.

input_dim = 1
hidden_dim = 32
num_layers = 2
output_dim = 1


# Here we define our model as a class
class LSTM(nn.Module):
  def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
    super(LSTM, self).__init__()
    self.hidden_dim = hidden_dim
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
    self.fc = nn.Linear(hidden_dim, output_dim)

  def forward(self, x):
    #Initialize hidden layer and cell state
    h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()
    c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()

    # We need to detach as we are doing truncated backpropagation through time (BPTT)
    # If we don't, we'll backprop all the way to the start even after going through another batch
    out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))


    # Index hidden state of last time step
    # out.size() --> 100, 32, 100
    # out[:, -1, :] --> 100, 100 --> just want last time step hidden states! 
    out = self.fc(out[:, -1, :])

    # For binomial Classification
    m = torch.sigmoid(out)

    return m

model = LSTM(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers)
loss = nn.BCELoss()

optimiser = torch.optim.Adam(model.parameters(), lr=0.00001, weight_decay=0.00006)

num_epochs = 100

# Number of steps to unroll
seq_dim =look_back-1  

for t in range(num_epochs):
    y_train_class = model(x_train)

    output = loss(y_train_class, y_train)

    # Zero out gradient, else they will accumulate between epochs
    optimiser.zero_grad(set_to_none=True)

    # Backward pass
    output.backward()

    # Update parameters
    optimiser.step()

This is an example of what the result looks like

This code is initially from kaggle, I edited them for classification. Please, can you tell me what I am doing wrong?

EDIT 1: Add dataloader from torch.utils.data import DataLoader from torch.utils.data import TensorDataset

x_train = torch.from_numpy(x_train).type(torch.Tensor)
y_train = torch.from_numpy(y_train).type(torch.Tensor)
x_test = torch.from_numpy(x_test).type(torch.Tensor)
y_test = torch.from_numpy(y_test).type(torch.Tensor)

train_dataloader = DataLoader(TensorDataset(x_train, y_train), batch_size=128, shuffle=True)
test_dataloader = DataLoader(TensorDataset(x_test, y_test), batch_size=128, shuffle=True)

I realized I had forgotten to inverse the transformation before checking the result. When I did that, I got different values from classification, however all values are in the scale of 0.001-0.009, so when I round them, the result is same. Label 0 for all classification.

lstm pytorch classification machine-learning python
1 Answers

A common phenomenon in NN training is that they will initially converge to a very naive solution to the problem where they output a constant prediction that minimizes the error on the training data. My guess is that in your training data, the ratio between 0 and 1 classes is close to 0.5423. Depending on whether your model is of sufficient complexity, it might learn to make more specific predictions based on the input when given more learning steps.

While increasing the number of epochs could help, there is something better you can do with your current setup. Currently, you are only performing a single optimizer step per epoch. Typically, you would want a step per batch and loop over your data in (mini)batches of, say, 32 inputs for example. To do this, it would be best to use a DataLoader where you can define a batch size, and loop over the dataloader inside your epoch loop similar to this example.

user_141474890
Related