Is there a way to restart CUDA GPU within python ? or release it so that another script can get access to it ?
I sometimes get an error using the GPU in python, and the only solution to get access to the GPU again is to restart my Jupyter notebook.
PS: I am using the GPU for some computation using pytorch.
the particular error I get is:
RuntimeError: CUDA error: device-side assert triggered
I have seen https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3 and many others discussing this problem. My main issue is how to continue using the GPU session without restarting the Jupyter kernel.
i.e. I would like to catch this error which I do with a try/except and move on since I am training multiple models one after the other so i cannot restart to solve my problem.
EDIT restarting the kernel seems to be the accepted naive solution see:
-https://forums.fast.ai/t/how-to-free-up-gpu-memory-in-pytorch-0-2-x/9256/2 -https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3 where she mentions "Working on Kaggle? Here’s why even after following the above steps, you are still struggling"