Resizing PyTorch tensor with grad to smaller size

user_4428725 0
Viewed 1392

I am trying to downsize a tensor from let's say (3,3) to (1, 1), but I want to keep the original tensor:

import torch

a = torch.rand(3, 3)
a_copy = a.clone()
a_copy.resize_(1, 1)

I need requires_grad=True in my initial tensor but PyTorch forbids this me from trying to resize the copy:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone()
a_copy.resize_(1, 1)

Throws an error:

Traceback (most recent call last):
  File "pytorch_test.py", line 7, in <module>
    a_copy.resize_(1, 1)
RuntimeError: cannot resize variables that require grad

Clone and Detach

I tried to .clone() and .detach() as well:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone().detach()

with torch.no_grad():
    a_copy.resize_(1, 1)

which gives this error instead:

Traceback (most recent call last):
  File "pytorch_test.py", line 14, in <module>
    a_copy.resize_(1, 1)
RuntimeError: set_sizes_contiguous is not allowed on a Tensor created from .data or .detach().
If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage_offset)
without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.
For example, change:
    x.data.set_(y)
to:
    with torch.no_grad():
        x.set_(y)

This behaviour had been stated in the docs and #15070.

With no_grad()

So, following what they said in the error message, I removed .detach() and used no_grad() instead:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone()

with torch.no_grad():
    a_copy.resize_(1, 1)

But it still gives me an error about grad:

Traceback (most recent call last):
  File "pytorch_test.py", line 21, in <module>
    a_copy.resize_(1, 1)
RuntimeError: cannot resize variables that require grad

Similar questions

I have looked at Resize PyTorch Tensor but it the tensor in that example retains all original values. I have also looked at Pytorch preferred way to copy a tensor which is the method I am using to copy the tensor.

I am using PyTorch version 1.4.0

pytorch python
2 Answers

There is a narrow() function:

def samestorage(x,y):
    if x.storage().data_ptr()==y.storage().data_ptr():
        print("same storage")
    else:
        print("different storage")
def contiguous(y):
    if True==y.is_contiguous():
        print("contiguous")
    else:
        print("non contiguous")
# narrow => same storage contiguous tensors
import torch
x = torch.randn(3, 3, requires_grad=True)
y = x.narrow(0, 1, 2) #dim, start, len  
print(x)
print(y)
contiguous(y)
samestorage(x,y)

Out:

tensor([[ 1.1383, -1.2937,  0.8451],
        [ 0.0151,  0.8608,  1.4623],
        [ 0.8490, -0.0870, -0.0254]], requires_grad=True)
tensor([[ 0.0151,  0.8608,  1.4623],
        [ 0.8490, -0.0870, -0.0254]], grad_fn=<SliceBackward>)
contiguous
same storage
user_58849550

I think you should first detach and then clone:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.detach().clone()
a_copy.resize_(1, 1)

Note: a.detach() returns a new tensor detached from the current graph (it doesn't detach a itself from the graph as a.detach_() does). But because it shares storage with a, you also should clone it. In this way, whatever you do with the a_copy won't affect a. However I am not sure why a.detach().clone() works but a.clone().detach() gives error.

Edit:

The following code also works (which is probably a better solution):

a = torch.rand(3, 3, requires_grad=True)

with torch.no_grad():
    a_copy = a.clone()
    a_copy.resize_(1, 1)
user_29453570
Related