I set up a g4dn.4xlarge instance, with 64 GiB memory, running an Amazon Machine Image Linux 2 with Tesla drivers (option 1 in this tutorial. I install and run Stable Diffusion. Text-to-image works well. When I run image-to-image, I get this error, showing that the GPU only has 14.76 GiB:
RuntimeError: CUDA out of memory. Tried to allocate 6.10 GiB (GPU 0; 14.76 GiB total capacity; 10.70 GiB already allocated; 1.24 GiB free; 12.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
I run nvidia-smi and see that the GPU only has around 15 GiB:
$ nvidia-smi
Wed Sep 21 09:38:16 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.119.01 Driver Version: 450.119.01 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 38C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
How can I tell CUDA to access the full 64 GiB available in the instance? Or how can I set up a virtual machine where CUDA uses all the available memory?