I built a django API which calls in some routes python functions which use GPU and cuda. When I call these routes the API always crash without an error message.
I was thinking it was caused by a lack of RAM so I increased the memory on my server but it still doesn't work.
Because there is no error message, I added a lot of print in my code until finding the problem always happend when the python code use cuda.
Here is the code which make crash the API :
from training.tacotron2_model import Tacotron2
model = Tacotron2().cuda()
I tried different things and this one worked perfectly :
model = Tacotron2()
But it doesn't use GPU so that's not what I want, It just demonstrates the problem is only with cuda.
What is the most interesting is that if I run both lines (using cuda) in a python terminal it perfectly works. So why doesn't it work when it's called by my django API ? I can't understand and have this problem with many routes since few days
I specify that I use a Nvidia Tesla T4 and that cuda is right installed. I can see it by running nvcc --version
or
import torch
torch.cuda.is_available()
and I got True