I'm interested in running multiple Pytorch models in a single GPU, more precisely YOLOv5-small in a single 3090. However, I have a problem when loading several models as the CPU RAM runs out of memory and I want to run inference in the GPU.
First I tried loading the architecture by the default way:
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
model = model.to('cuda')
but whenever the model is loaded in the GPU, both the CPU RAM and GPU RAM increase by a huge amount.
Then I tried to load a torchvision model to see if there was a problem in the yolov5 implementation:
model = torch.hub.load('pytorch/vision:v0.9.0', 'deeplabv3_resnet101', pretrained=True)
In this case, loading the model to CPU increased in a reasonable amount the CPU RAM. But when the model is sent to the GPU with model.to('cuda'), I obtain the same performance as in the first scenario.
I have tried using gc.collect() and del model once it has send to gpu, but I haven't managed solving this issue.
I am using: CPU: AMD Ryzen 9 3900XT 12-Core Processor. 32GB of RAM memory. GPU: GeForce RTX 3090.
Hope someone knows what exactly happens. Thanks in advance.