hello~ I am very confused with this situation.
first, both my tf and pytorch can detect my gpu (use torch.cuda,is_available()) but my model which runs just fine on gpus just few days before can only run on cpus today. it seems pytorch and tf skip passing model to gpu directly.
second, I have test in python interactive mode with:
import torch
x = torch.randn(10000,1000).cuda()
this line worked fine, and when I type
x.device
python shows me that x is on gpu device index 0 but at the same time. NO GPU MEMORY in use in nvidia-smi
third, when I monitor my gpu states with
watch -n 1 nvidia-smi
I found temperature or power of my gpus does not vary at all for a long while.
any help will be appreciated!!