.grad variable suddenly becomes not a number (nan)

Viewed 29

All the Grad variables w.. and b.. become "None" or nan

Do you understand because this code doesn't work?

I was trying to simulate two hidden layers with a gradient descent.

There's something wrong with the "predicted" variable?

valoreDiInput = 3
target = 50
learning_rate = 0.01
num_epochs= 100

w01 = torch.tensor(1., requires_grad=True) # first layer
b01 = torch.tensor(1., requires_grad=True) 
w11 = torch.tensor(1., requires_grad=True) # second layer
b11 = torch.tensor(1., requires_grad=True)

for epoch in range(num_epochs):
    predicted = (valoreDiInput * w01 + b01) * w11 + b11
    loss = (predicted - target)**2 # positive loss
    loss.backward()
    
    with torch.no_grad():
        w01 -= w01.grad * learning_rate
        b01 -= b01.grad * learning_rate
        w01.grad.zero_()
        b01.grad.zero_()
        w11 -= w11.grad * learning_rate
        b11 -= b11.grad * learning_rate
        w11.grad.zero_()
        b11.grad.zero_()

    if epoch % 10==0:
        print(f'Epoch {epoch}  Loss {loss.item():.6f} w01 {w01.item():.6f} b01 {b01.item():.6f}')
0 Answers
Related