cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:51

user_8748450 0
Viewed 2873

I start to run TensorFlow application on terminal. At the same time, when I start Pytorch application on another terminal I get error

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51  error=3 : initialization error
terminate called after throwing an instance of 'std::runtime_error'
what():  cuda runtime error (3) : initialization error at /pytorch /aten/src/THC/THCGeneral.cpp:51

Both Pytorch TensorFlow and running on different virtualenvs

My Environment -Ubuntu 18.04

-GPU GeForce GTX 1060

-Pytorch env (torch==1.1.0, torchvision==0.2.0)

-Tensorflow env (tensorflow-gpu==1.15.0)

Pytorch application was running smoothly before starting TensorFlow application

I stop the TensorFlow application and check 

>>> torch.cuda.device_count()
0
>>> torch.cuda.is_available()
False

But error not goes back

pytorch runtime-error nvidia python-3.x
1 Answers

I had the same problem with pytorch training script crashing, after waking up the Ubuntu from sleep mode. Torch was not able to detect the GPU. It seems that the Cuda driver has a problem restoring the active context after the wakeup.

Rebooting the system fixed the issue.

I was running pytorch in "conda env".

user_143790190
Related