I'm trying to run a piece of code on Pytorch, but I get the error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
I've narrowed down the issue to be a missmatch of CUDA versions. My machine has 2 GPUs:
a GeForce GTX 650 (compute capability 3.0) and a Tesla K40c (compute capability 3.5). I've checked the compute capabilities here: https://developer.nvidia.com/cuda-gpus.
My nvidia-smi command gives the following:
nvidia-smi output (Driver Version: 470.57.02 & CUDA Version: 11.4)
While my nvcc -V command gives the following:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
The 10.1 version exists because I tried to install that CUDA version, specifically, following the instructions elsewhere (for example: https://medium.com/@anarmammadli/how-to-install-cuda-10-2-cudnn-7-6-5-and-samples-on-ubuntu-18-04-2493124478ca)
Also, I've installed cudatoolkit with conda, and so on my conda list I have the following entry:
...
cudatoolkit 10.1.243 h6bb024c_0
...
In accordance with https://github.com/moi90/pytorch_compute_capabilities/blob/main/table.md I've also installed the 1.8.0 PyTorch version.
However, in Python 3.7.11:
Python 3.7.11 (default, Jul 27 2021, 14:32:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1'
>>> torch.version.cuda
'10.1'
>>> torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'compute_37']
>>> torch.cuda.is_available()
True
I don't have sm_35 available, which I need to use the Tesla K40. I believe this is the reason why I keep getting the CUDA error: no kernel image is available for execution on the device error. I've also tried all the above for version 10.2 of CUDA, same result.