CUDA Runtime Error: Which Cuda version is compatible to run NER task using BERT-NER

user_1872999 0
Viewed 841

I have setup all the requirement packages installed on my VM and i found no nvidia GPU driver installed, In the requirements doesn't have nvidia GPU driver installation instructions, I want to know which cuda version and it compatible nvidia driver which needs too resolve the below error.

Github link: github

Error logs:

  File "run_ner.py", line 594, in <module>
    main()
  File "run_ner.py", line 489, in main
    loss = model(input_ids, segment_ids, input_mask, label_ids,valid_ids,l_mask)
  File "/home/pt3_gcp/BERT-NER/ber_ner/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "run_ner.py", line 35, in forward
    valid_output = torch.zeros(batch_size,max_len,feat_dim,dtype=torch.float32,device='cuda')
  File "/home/pt3_gcp/BERT-NER/ber_ner/lib/python3.7/site-packages/torch/cuda/__init__.py", line 178, in _lazy_init
    _check_driver()
  File "/home/pt3_gcp/BERT-NER/ber_ner/lib/python3.7/site-packages/torch/cuda/__init__.py", line 99, in _check_driver
    http://www.nvidia.com/Download/index.aspx""")
AssertionError: 
**Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx
**

After installing latest cuda version from the following link, cuda I got the following error,

06/04/2020 07:38:40 - INFO - __main__ -   ***** Running training *****
06/04/2020 07:38:40 - INFO - __main__ -     Num examples = 14041
06/04/2020 07:38:40 - INFO - __main__ -     Batch size = 32
06/04/2020 07:38:40 - INFO - __main__ -     Num steps = 2190
Epoch:   0%|                                                                                 | 0/5 [00:00<?, ?it/sTHCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=50 error=38 : no CUDA-capable device is detectedt/s]
Traceback (most recent call last):
  File "run_ner.py", line 594, in <module>
    main()
  File "run_ner.py", line 489, in main
    loss = model(input_ids, segment_ids, input_mask, label_ids,valid_ids,l_mask)
  File "/home/pt3_gcp/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "run_ner.py", line 35, in forward
    valid_output = torch.zeros(batch_size,max_len,feat_dim,dtype=torch.float32,device='cuda')
  File "/home/pt3_gcp/.local/lib/python3.7/site-packages/torch/cuda/__init__.py", line 179, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50
named-entity-recognition bert-language-model huggingface-transformers pytorch
1 Answers

I had the same issue for some time ago. The following commands fixed for me!

It is a problem if you have multiple installation and likely you have now since you tried many things. Remove basically everything

sudo apt-get purge nvidia-*
sudo apt-get remove nvidia-cuda-toolkit
sudo apt autoremove --purge cuda-10-0 // you might have a different version, check it git cuda --version

also delete the existing files in your user base

rm -rf /usr/local/cuda* // anything related to cuda
rm -rf /usr/local/nvidia* // anything related to nvidia

Now, finally a fresh install

sudo apt-get update // update your packages

sudo apt search nvidia-driver  // to get the latest version of the driver. After finding out the latest version, install it with

sudo apt install nvidia-driver-450 (or any other number, depending on the latest version)

After installing it, you must reboot!

sudo reboot

When you are back, nvidia-smi should work and also your gpus

user_80105680
Related