How to get gradient (dL/dw) during training in Pytorch?

user_12030208 0
Viewed 38
class ConvNet(torch.nn.Module):
 def __init__(self):

   super(ConvNet, self).__init__()
   self.conv1 = torch.nn.Conv2d(1, 6, 5)
   self.pool = torch.nn.MaxPool2d(2, 2)
   self.conv2 = torch.nn.Conv2d(6, 16, 5)
   self.fc1 = torch.nn.Linear(16 * 4 * 4, 62)

 def forward(self, x):
   x = self.pool(F.relu(self.conv1(x)))
   x = self.pool(F.relu(self.conv2(x)))
   x = x.view(-1, 16 * 4 * 4)
   x = self.fc1(x)
   return x

for x, y in loader:
  x, y = x.to(device), y.to(device)
  optimizer.zero_grad()
  loss = torch.nn.CrossEntropyLoss()(model(x), y)
  loss.backward()

  print(Convnet().conv1.weight.grad)

  optimizer.step()

I tried Convnet().conv1.weight.grad, but it gives None output. What are the other options to print gradient in Pytorch?

pytorch machine-learning
1 Answers

Okay, a few things to note here:

  1. I'm assuming you have already instantiated/initialized your ConvNet class with an object called model. (model = ConvNet())
  2. The way you're accessing the model's weight gradients is correct, however, you're using the wrong object to access these weights. Specifically, you're supposed to use the instantiated running model to access these gradients, which is the model object you instantiated. When you use ConvNet().conv1.weight.grad, you're creating a new instance of the class ConvNet() on every call, and none of these instances were used to process your data x, hence they all give None for gradients.
  3. Based on the above points, the correct way to access the gradients is to use your instaniated model which you've used to process your data, which is: model.conv1.weight.grad
  4. Side note; you might want to use torch's functional API to find the loss as it's more readable: loss = F.cross_entropy(model(x), y)
user_144920010
Related