Why do we need to pass the gradient parameter to the backward function in PyTorch?

user_11732239 0
Viewed 412

According to the docs, when we call the backward function to the tensor if the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying gradient.

import torch
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([1.,1.])) 

print(a.grad)

Output: tensor([20., 20.])

Now scaling the external gradient:

a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([2.,2.])) #modified

print(a.grad)

Output: tensor([40., 40.])

So, passing the gradient argument to backward seems to scale the gradients.
Also, by default F.backward() is F.backward(gradient=torch.Tensor([1.]))

Apart from scaling the grad value how does the gradient parameter passed to the backward function helps to compute the derivatives when we have a non-scalar tensor?
Why can't PyTorch calculate the derivative implicitly without asking explicit gradient parameter as it did for the scalar tensor?

pytorch
1 Answers

It's because PyTorch is calculating the jacobian product. In case of scalar value, .backward() w/o parameters is equivalent to .backward(torch.tensor(1.0)).

That's why you need to provide the tensor with which you want to calculate the product. Read more about automatic differentiation.

user_62108070
Related