I have previously had CUDA 9.x running on this Win 10 64-bit Home system (targeting 1080Ti card), but need to update to CUDA 10.0 for TensorFlow 2. I initially thought TF2 was OK with CUDA 10.1 and so first installed 10.1 and only later found out that it must be CUDA 10.
Can't get it to work...
To test TF, I ran this to validate the installation(Jupyter notebook via Anaconda - freshly built TF2 environment)
import tensforflow as tf
print(tf.reduce_sum(tf.random.normal([1000, 1000])))
I get this error in the basic Python test
InternalError: cudaGetDevice() failed. Status: cudaGetErrorString symbol not found
This suggests that a key file cannot be found, but I can't work out the root cause - and there are very few hits on that error info, none of which helped me.
Current Config
CUDA 10.0 installed Nvidia driver 436.48 game ready driver
Potential issues & resolution actions so far
Obviously none of them have fixed things
- Old CUDA installations - 9.0, 9.1, 10.0, 10.1: all except 10.0 uninstalled and PC rebooted; 10.0 installer then run again
- Updating cudnn files: tried 1st with the originals and then cudnn files v7.6.3.30 copied to bin, include, lib as appropriate
- Switch from game ready driver to "Studio" driver
- Check all environment variables - removed everything that referred to CUDA != 10.0
- Update renamed nvcuda.dll to .old in system32 and reran the CUDA 10.0 installer... a new nvcuda was not produced.
- Update 2 I found nvcuda64.dll v 10.0.132 in the driver store and replaced nvcuda.dll in system32 with it; after reboot, nvidia-smi now reports no CUDA version at all :(
Known Oddities
[superseded by Update 2] nvidia-smi.exe reports CUDA 10.1 (yes, it is available on my Win 10) - but checking through the registry I can't find anything to suggest CUDA 10.1 is lingering there...Update Found it in C:\Windows\System32
Despite uninstalls, I still have CudaXYZWizardsPackage in the registry under the key Computer\HKEY_USERS.DEFAULT\Software\Microsoft\VisualStudio\14.0_Config\InstalledProducts with XYZ = 90, 91, 100, 101 - but I doubt this is the issue for TF in Python ;) Update there is nothing in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\NVIDIA except for 10.0 so just orphan reg entries.
Other info
- Before doing all the uninstalls etc. I did successfully build and run the Nvidia sample clock project in VS 2017 so the basics were OK (then)
Questions
- How can I completely remove all trace of CUDA to start again from a clean slate?
- How could I diagnose such issues in future to work out where the issue is/what to do
- Can this particular issue be resolved more simply?
- (New) Where can I get nvcuda.dll 10.0 to replace in system32? - Answer one possibility is from C:\Windows\System32\DriverStore\FileRepository