Why doesn't pytorch allow inplace operations on leaf variables?

user_6411761 0
Viewed 3518

So if I run this code in Pytorch:

x = torch.ones(2,2, requires_grad=True)
x.add_(1)

I will get the error:

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

I understand that Pytorch does not allow inplace operations on leaf variables and I also know that there are ways to get around this restrictions. What I don't understand is the philosophy behind this rule. Why is it wrong to change a leaf variable with inplace operations?

pytorch python
2 Answers

As I understand it, any time you do a non-traditional operation on a tensor that was initialized with requires_grad=True, Pytorch throws an error to make sure it was intentional. For example, you normally would only update a weight tensor using optimizer.step().

For another example, I ran into this issue when trying to update the values in a backprop-able tensor during network initialization.

self.weight_layer = nn.Parameter(data=torch.zeros(seq_length), requires_grad=True)
self.weight_layer[true_ids == 1] = -1.2

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

The problem is that, because requires_grad=True, the network doesn't know that I'm still initializing the values. If this is what you are trying to do, wrapping the update in a torch.no_grad block is one solution:

with torch.no_grad()
    self.weight_layer = nn.Parameter(data=torch.zeros(seq_length), requires_grad=True)
    self.weight_layer[true_ids == 1] = -1.2

Otherwise, you could just set requires_grad=True after you finish initializing the Tensor:

self.weight_layer = nn.Parameter(data=torch.zeros(seq_length))
self.weight_layer[true_ids == 1] = -1.2
self.weight_layer.requires_grad = True
user_92701810

The simple answer to this is that, once autograd creates the graph, and we have created our tensors which are a input to the graph, autograd will construct and track each operation on your created tensor. Now, since this tensor has requires_grad=True, and let's say I do a weight update after loss.backward(), autograd is probably considering as a part of the already created graph, which requires gradients. This leads to the RuntimeError: a leaf Variable that requires grad is being used in an in-place operation. Autograd is confused that why is a tensor which is a part of the computational graph being used outside/being used for some other operations/initialization of it inplace and this is an an problem

if we simply place the code under a with torch.no_grad(): code here we disable the gradients, and hence there is essentially it signals to autograd that this operation is not a part of our dynamic graph updates.

PS: I will expand this answer by writing a blog of this on Medium

user_118154040
Related