In Pytorch I am optimizing a model. If I run the following code, nvidia-smi shows that I am using 850MiB / 7979MiB of memory on my gpu. Why would this be the case?
with torch.no_grad(): A = nn.Conv2d(10,10,3).cuda()
I imagine there are some overhead or a default allocation size specified somewhere but I could not find such documentation. I do recall that tensorflow had a setting to limit the amount of memory allocated.
The convolution does not occupy that much memory. You can verify this with torch.cuda.memory_allocated, which shows the memory that is occupied by all tensors in bytes:
torch.cuda.memory_allocated() # => 0
A = nn.Conv2d(10,10,3).cuda()
torch.cuda.memory_allocated() # => 4608
The convolution only uses 4608 bytes.
nvidia-smi shows higher memory usage for two separate reasons.
PyTorch uses a caching memory allocator, meaning that it holds onto more memory than necessary to avoid device synchronisations.
From PyTorch CUDA Semantics - Memory Management:
PyTorch uses a caching memory allocator to speed up memory allocations. This allows fast memory deallocation without device synchronizations. However, the unused memory managed by the allocator will still show as if used in
nvidia-smi. You can usememory_allocated()andmax_memory_allocated()to monitor memory occupied by tensors, and usememory_reserved()andmax_memory_reserved()to monitor the total amount of memory managed by the caching allocator.
When CUDA is first initialised, it creates a context that manages the control of the device. Most notably, the context contains the code of all the different CUDA kernels, of which PyTorch has a large number. The size of the context also varies across different GPU architectures. Some details are discussed in Issue #20532 - Couple hundred MB are taken just by initializing cuda .
The memory you are observing is almost exclusively attributed to the CUDA context.