what does pytorch do for creating tensor from numpy

user_7672954 0
Viewed 212

I am interested in what torch have done when I call torch.from_numpy. As the name indicates, it seems that PyTorch creates a Tensor instance and allocates the memory for copying the content from numpy ndarray to itself. But how does PyTorch do the memcpy work and what else has PyTorch done in the background? It seems the implementation of tensor is in autograd. But I have no idea which part should I look for.

I have the question because I found it is really fast for constructing a tensor from numpy. And it is even fast than creating a tensor directly

a = np.random.randn(100,100)
%timeit torch.from_numpy(a)
759 ns ± 7.53 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
%timeit torch.randn(100,100)
61 µs ± 2.46 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit torch.zeros(100,100)
3.1 µs ± 136 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
tensor pytorch numpy-ndarray numpy python
1 Answers

The documentation explains that

The returned tensor and ndarray share the same memory. Modifications to the tensor will be reflected in the ndarray and vice versa. The returned tensor is not resizable.

These sentences implies that there is no memcopy involved (otherwise modifications would not be reflected in one aonther). That is why the operation is so fast : pytorch merely creates a pointer to the numpy array underlying data, and "assigns" this pointer to a tensor. This function does not allocate or copy any memory at all. Therefore, from_numpy is just duplicating a pointer (which is an integer number) and probably performing a few checks.

What is important to remember is really that the underlying memories are shared and therefore the tensor and the numpy array modify one another, and you should use clone or copy to perform a clean deep copy and get rid of this behavior (if you need to), like

b = torch.from_numpy(a).clone()
user_109357170
Related