My computer is in Ubuntu 22.04 LTS, with Nvidia GeForce 2080Ti. And I am training my CNNs in one GPU. Python 3.9.0 and Pytorch 1.11.0+cu102.
When training, My code allocate about 8000-9000MB of GPU memory, in mixed precision way. And GPU-Util bouncing from 0 to 80 around, then drop back to 0. And my cores are active, too. So it might not be I/O bottleneck? But as normal, high GPU-Util keep around 80-100 is normal.All these are refers to nnUNet expected epoch times here.
As a result of bouncing GPU-Util(might not be), my training took about half an hour in each epoch. It seems not right.
For my whole framework needs to load teacher model, and 3 sub-networks for assistant. The parameters number maybe very huge. And I run torchsummary, and get this.
Assume that batch_size is 2.
- GAN(totally 3 generators, 1 channel)
================================================================
Total params: 6,420,441
Trainable params: 6,420,441
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 8.00
Forward/backward pass size (MB): 17320.00
Params size (MB): 24.49
Estimated Total Size (MB): 17352.49
----------------------------------------------------------------
- Teacher network(4 channels, student network keeps the same)
================================================================
Total params: 16,137,152
Trainable params: 16,137,152
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 32.00
Forward/backward pass size (MB): 5223.15
Params size (MB): 61.56
Estimated Total Size (MB): 5316.71
----------------------------------------------------------------
I google and search some solutions, suggesting me to make batch size bigger. For loading and processing data took CPU too much times while GPU just finishes calculating in very little second.
So, in such 2080Ti GPU(total 11000MiB), I only can run batch_size is 2 training, bigger one cause CUDAError.
Now I have no idea, here is my question:
- How to improve GPU-Util
- Is it still ok with this total params? If so, how to reduce some unnecessary GPU memory occupying when computing?
Thanks a lot!