I am kinda new to the pytorch, now struggling with a classification problem

Mercury460 0
Viewed 48

I built a very simple structure

class classifier (nn.Module):
def __init__(self):
    super().__init__()
    self.classify = nn.Sequential(
        nn.Linear(166,80),
        nn.Tanh(),
        nn.Linear(80,40),
        nn.Tanh(),
        nn.Linear(40,1),
        nn.Softmax()
    )
def forward (self, x):
    pred = self.classify(x)
    return pred
model = classifier()

The loss function and optimizer are defined as

criteria = nn.BCEWithLogitsLoss()
iteration = 1000
learning_rate = 0.1
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

and here is the training and evaluation section

for epoch in range (iteration):
  model.train()
  y_pred = model(x_train)
  loss = criteria(y_pred,y_train)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()


  model.eval()
  with torch.inference_mode():
      test_pred = model(x_test)
      test_loss = criteria(test_pred, y_test)

  if epoch % 100 == 0:
      print(loss)
      print(test_loss)

I received the same loss values, and by debugging, I found that the weights were not being updated.

classification pytorch
2 Answers

The problem is in the network architecture: you are using a Softmax layer on a single valued output at the end. As per the definition of the softmax function, for a output vector x, we have, for index i:

softmax(x_i) = e^{x_i} / sum_j (e^{x_j})

Here, you only have a single valued output. Due to this, the output of your neural network is always 1, irrespective of the inputs or the weights. To fix this, remove the Softmax layer at the end. An activation function like Sigmoid might be more appropriate, and in fact you are already applying this when using the BCEWithLogitsLoss.

GoodDeeds0

The problem lies here

y_pred = model(x_train)
  loss = criteria(y_pred,y_train)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

after loss is calculated, you are clearing the gradients by doing optimizer.zero_grad()

the ideal case should be:


  optimizer.zero_grad()
  y_pred = model(x_train)
  loss = criteria(y_pred,y_train)
  loss.backward()
  optimizer.step()
harshraj220
Related