Problem in basic parameter initialization

Xianglong Chen 0
Viewed 26

Thanks for your time~ after I run the code, the grad is always zero and the loss is not updating.(I guess it's because the weights is initialized all 0's, but I don't know how to fix it) The code is a basic neural network:

class Model(torch.nn.Module):       #class
    def __init__(self):
        super(Model, self).__init__()        
        self.linear1 = torch.nn.Linear(8,6)        
        self.linear2 = torch.nn.Linear(6,4)        
        self.linear3 = torch.nn.Linear(4,1)    
        self.sigmoid = torch.nn.Sigmoid()
        
    def forward(self, x):
        x = self.sigmoid(self.linear1(x))
        x = self.sigmoid(self.linear2(x))
        x = self.sigmoid(self.linear3(x))
        x = F.softmax(x, dim=1)
        return x
   
model = Model()   #model
criterion = torch.nn.BCELoss(size_average = False)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(1000):     # training
    y_pred = model(X_train.float())
    loss = criterion(y_pred, y_train.float())
    print(epoch, loss.item())
    print([x.grad for x in optimizer.param_groups[0]['params']])
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

And I get the 0s grad:

enter image description here

initialization parameters pytorch python
1 Answers

I think your forgot to apply the backpropagation. Adding loss.backward() just before your print statetements will do the trick (compute the accumulated gradients and store them in x.grad). Note that by default your weights are not initialized to 0 here. The default initialization for linear layers are here.

Valentin Goldité0
Related