Gradient Descent Reassignment in Pytorch

user_18193186 0
Viewed 134

I was following a series of tutorial on youtube about deep learning, and I encountered a problem which really confuses me.

X = torch.tensor([1,2,3,4], dtype = torch.float32)
Y = torch.tensor([2,4,6,8], dtype = torch.float32)

w = torch.tensor(0.0, dtype = torch.float32, requires_grad=True)

def forward(x):
  return w*x;

def loss(y, y_predicted):
  return ((y-y_predicted)**2).mean()

print(f'Prediction before training: f(5) = {forward(5):.3f}')


learning_rate= 0.01
epoch = 20
for i in range(epoch):
  y_pred = forward(X)
  l = loss(Y, y_pred)
  l.backward()
  with torch.no_grad():
    w = w - learning_rate * w.grad
    # (w -= learning_rate * w.grad) # would not cause error in the following line


  w.grad.zero_() #error : 'NoneType' object has no attribute 'zero_'
  if i % 1 ==0:
    print(f'weight : {w}, loss : {l}')

I really wonder the difference between "w = w - learning_rate * w.grad" and "w -= learning_rate * w.grad" cause these two are the same in my expericence. Thanks!

pytorch deep-learning python
2 Answers

As pointed out in the comment, the problem is in how Pytorch computes/stores gradients. In fact,

w-= learning_rate * w.grad

is an in-place operation, which will make w keep its initial properties (the requires_grad=True). Usually in Pytorch, we avoid in-place operations as it may break the computational graph used by Autograd (see Pytorch Forum Post).

But for you, this:

w = w - learning_rate * w.grad

is not in-place. Thus, w is assigned to a new copy, and because of the torch.no_grad() statement, this copy won't have a .grad attribute.

user_132150970

Although operators are working in python with a specific behavior, this is not always true for the pytorch framework. For the vanilla python interpretation, we could watch that the subtraction assignment of A -= B is equivalent for A = A - B syntax. Additionally, python operators can be used every time you want to assign a value to a variable. In fact, we call this syntactic sugar in python, and it can be used to simplify your code. However, things work differently for the pytorch framework.

In your case, the new w is assigned after l.backward() being propagated. Hence, it has no grad value. Moreover, the w assignment has no requires_grad because of the torch.no_grad() condition.

Now, let's take into account your example:

w = w - learning_rate * w.grad

In the pytorch framework, it is equivalent to:

w[...] = w - learning_rate * w.grad

Although w[...] = w - learning_rate * w.grad and w -= learning_rate * w.grad might seem equivalent, they don't follow the same structure. Hence, they promote different behaviors. On the one hand, the w -= learning_rate * w.grad expression is considered as an in-place operation. Meaning that it changes directly the content of a given tensor (e.g., w[...]) without making a copy. On the other hand, the requires_grad=True attribute is making w keep its initial properties. Therefore, not updating its state.

user_23719870
Related