w -= dw vs w = w - dw in torch

user_2171154 0
Viewed 39

I discovered a torch behavior by accident today that w = w - dw does not strictly equal to w -= dw, simple example attached below:

x_input_, y_gt_ = datasets.make_regression(n_samples=100, n_features=1)
    # Model
    x_input = torch.from_numpy(x_input_)
    w = torch.tensor(1, requires_grad=True, dtype=torch.float32)

    x_input = torch.from_numpy(x_input_)
    y_gt = torch.unsqueeze(torch.from_numpy(y_gt_), dim=1)

    lr = 0.01
    for iter in range(3):
        y_pred = w * x_input
        loss = ((y_gt - y_pred) ** 2).mean()
        loss.backward()
        with torch.no_grad():
            dw = w.grad * lr
            print(w)
            w -= dw
            print(w)

which outputs (expected)

tensor(1., requires_grad=True)
tensor(1.0413, requires_grad=True)
tensor(1.0413, requires_grad=True)
tensor(1.1230, requires_grad=True)
tensor(1.1230, requires_grad=True)
tensor(1.2431, requires_grad=True)

however if I replace w -= dw as w = w - dw, it will say that gradient is no longer attached

tensor(1., requires_grad=True)
tensor(3.1186)

I am curious what might have caused this?

torch python
1 Answers

The expression w -= dw calls a special function on w that allows it to modify itself.

The expression w - dw creates a new object, and the w = assigns that new object in place of the old one.

Mark_Ransom_59870
Related