An error when calling tensor.backward() in pytorch may caused by grad_fn=<CopySlices>

user_18417835 0
Viewed 244

I'm facing a problem in pytorch backward() function, here's my code

c = torch.empty(4)

a = torch.tensor(2.,requires_grad=True)

b = torch.tensor(3.,requires_grad=True)

c[0] = a*2

#c[0]:tensor(4., grad_fn=<SelectBackward>)

#c:tensor([4.0000e+00, 3.1720e+00, 1.0469e-38, 9.2755e-39], grad_fn=<CopySlices>)

c[0].backward()

And that would just work fine, and I can get the corret answer a.grad==tensor(2.), but if I excude the folloing code after above code:

c[1] = b*2

c[1].backward()

it would cause the follow erroe:

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

But if I writec[0].backward(retain_grad=True) in the first part, it would not cause the error.

Could someone tell me what intermediate results are freed during the backward. Is there some problem with CopySlices?

Thank you so much!

pytorch machine-learning python
1 Answers

To reduce memory usage, during the .backward() call, all the intermediary results are deleted when they are not needed anymore. Hence if you try to call .backward() again, the intermediary results don’t exist and the backward pass cannot be performed (and you get the error you see). You can call .backward(retain_graph=True) to make a backward pass that will not delete intermediary results, and so you will be able to call .backward() again. All but the last call to backward should have the retain_graph=True option.

c[0] = a*2

#c[0]:tensor(4., grad_fn=<SelectBackward>)

#c:tensor([4.0000e+00, 3.1720e+00, 1.0469e-38, 9.2755e-39], grad_fn=<CopySlices>)

c[0].backward(retain_graph=True)


c[1] = b*2

c[1].backward(retain_graph=True)
```
user_17714450
Related