When trying to use the GPU enabled tensorflow, I run into some error messages that seem to limit my ability to run my GPU despite many success messages in the log (see below). In terms of software, I am using an Anaconda Lite environment. In terms of hardware, I have an RTX 2080 gpu. Below are the packages installed on my environment.
I am using this tensorflow approved dependency combination:
Here is the important installs, but I attached the entire conda list below.
- TensorFlow-GPU 2.4.0
- Cudnn 8.0.5
- Cudatoolkit 11.0.3
absl-py 0.12.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
ca-certificates 2020.12.5 h5b45459_0 conda-forge
cachetools 4.2.1 pypi_0 pypi
certifi 2020.12.5 py37h03978a9_1 conda-forge
chardet 4.0.0 pypi_0 pypi
cudatoolkit 11.0.3 h3f58a73_8 conda-forge
cudnn 8.0.5.39 hfe7f257_1 conda-forge
cycler 0.10.0 pypi_0 pypi
filetype 1.0.7 pypi_0 pypi
flatbuffers 1.12 pypi_0 pypi
gast 0.3.3 pypi_0 pypi
google-auth 1.28.0 pypi_0 pypi
google-auth-oauthlib 0.4.4 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.32.0 pypi_0 pypi
h5py 2.10.0 pypi_0 pypi
idna 2.10 pypi_0 pypi
imageai 2.1.6 pypi_0 pypi
importlib-metadata 3.10.0 pypi_0 pypi
keras 2.4.3 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
keras-resnet 0.2.0 pypi_0 pypi
kiwisolver 1.3.1 pypi_0 pypi
markdown 3.3.4 pypi_0 pypi
matplotlib 3.3.2 pypi_0 pypi
numpy 1.19.3 pypi_0 pypi
oauthlib 3.1.0 pypi_0 pypi
opencv-python 4.5.1.48 pypi_0 pypi
openssl 1.1.1k h8ffe710_0 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
pillow 7.0.0 pypi_0 pypi
pip 21.0.1 py37haa95532_0
protobuf 3.15.7 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyparsing 2.4.7 pypi_0 pypi
python 3.7.6 h60c2a47_2
python-dateutil 2.8.1 pypi_0 pypi
python_abi 3.7 1_cp37m conda-forge
pyyaml 5.4.1 pypi_0 pypi
requests 2.25.1 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rsa 4.7.2 pypi_0 pypi
scipy 1.4.1 pypi_0 pypi
setuptools 52.0.0 py37haa95532_0
six 1.15.0 pypi_0 pypi
sqlite 3.35.3 h2bbff1b_0
tensorboard 2.4.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.0 pypi_0 pypi
tensorflow 2.4.0 pypi_0 pypi
tensorflow-estimator 2.4.0 pypi_0 pypi
tensorflow-gpu 2.4.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
typing-extensions 3.7.4.3 pypi_0 pypi
urllib3 1.26.4 pypi_0 pypi
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 1.0.1 pypi_0 pypi
wheel 0.36.2 pyhd3eb1b0_0
wincertstore 0.2 py37_0
wrapt 1.12.1 pypi_0 pypi
zipp 3.4.1 pypi_0 pypi
When executing code related to the ImageAI python package, I run into the following print outs. Note: there are alot of successful DLL loads, however, there are three important points of failure to note.
- Couldn't invoke ptxas.exe --version
- Call to CreateProcess failed. Error code: 2
- Not creating XLA devices, tf_xla_enable_xla_devices not set
Ptas: Tried to follow some other threads and came to the conclusion that this feature appears to be in development. Tensorflow 2.4.1 - Couldn't invoke ptxas.exe
CreateProcess: I have found threads with this error, but none in relation to tensorflow.
tf_xla_enable_xla_devices: This is a error that is OK to ignore, per this thread: https://github.com/tensorflow/tensorflow/issues/44683
Attached is the entire logging when a tensorflow script is called.
error:2021-04-04 19:16:49.100399: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
error:2021-04-04 19:16:50.654630: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
error:2021-04-04 19:16:50.655534: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
error:2021-04-04 19:16:50.685778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2080 computeCapability: 7.5
coreClock: 1.71GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.23GiB/s
2021-04-04 19:16:50.685800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
error:2021-04-04 19:16:50.696485: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-04 19:16:50.696504: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
error:2021-04-04 19:16:50.700890: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
error:2021-04-04 19:16:50.702964: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
error:2021-04-04 19:16:50.711373: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
error:2021-04-04 19:16:50.714875: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
error:2021-04-04 19:16:50.716294: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
error:2021-04-04 19:16:50.716353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
error:2021-04-04 19:16:50.716572: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-04 19:16:50.716942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2080 computeCapability: 7.5
coreClock: 1.71GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.23GiB/s
2021-04-04 19:16:50.716955: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
error:2021-04-04 19:16:50.716962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-04 19:16:50.716968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-04 19:16:50.716975: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-04 19:16:50.716979: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-04 19:16:50.716983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-04 19:16:50.716988: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-04 19:16:50.716992: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-04 19:16:50.717021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
error:2021-04-04 19:16:51.182297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-04 19:16:51.182316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-04-04 19:16:51.182320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
error:2021-04-04 19:16:51.182515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6599 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
error:2021-04-04 19:16:51.184721: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
error:2021-04-04 19:16:52.512582: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
error:2021-04-04 19:16:52.527832: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
error:2021-04-04 19:16:53.360829: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
2021-04-04 19:16:53.360847: W tensorflow/stream_executor/gpu/asm_compiler.cc:55] Couldn't invoke ptxas.exe --version
error:2021-04-04 19:16:53.369627: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
error:2021-04-04 19:16:53.369929: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Failed to launch ptxas
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
error:2021-04-04 19:16:53.375630: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
error:2021-04-04 19:16:53.381209: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
error:2021-04-04 19:16:53.386810: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
error:2021-04-04 19:16:53.391792: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
error:2021-04-04 19:16:53.396262: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
error:2021-04-04 19:16:53.401723: E tensorflow/core/platform/windows/subprocess.cc:283] Call to CreateProcess failed. Error code: 2
There is a whole bunch more of the same print out that I have omitted.