I have a source code that was built with tensorflow 2.3. However, I have multiple CUDAs (11.4, 10.1 and 9.0) installed in my machine, but the source code with the tensorflow 2.3 version was built to run with CUDA 9.0 or anything that contains libcudnn.so.7.
To make things more clear: my original (official) cuda version is 11.4 (once I do nvidia-smi I can see such a version). I believe it is installed in /usr/local/cuda.
The output of my nvidia-smi is
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro K620 Off | 00000000:01:00.0 On | N/A |
| 34% 44C P8 1W / 30W | 382MiB / 1994MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
If I list /usr/local I have as output
>> ls /usr/local
bin cuda cuda-10.1 cuda-9.0 etc games include lib lib64 man sbin share src
I followed this Tutorial that asks me to create an anaconda environment (which I believe it is a great idea, as I want to work with virtual environments only), and create two .sh files that are fired once I activate and deactivate the environment, which will tell the conda env which exact CUDA version I will use in that environment.
Below are the contents of ~/anaconda3/envs/my_train/etc/conda/activate.d/activate.sh as written in the Tutorial
ORIGINAL_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:/usr/local/cuda-9.0/extras/CUPTI/lib64:/lib/nccl/cuda-9:$LD_LIBRARY_PATH
And below are the contents of ~/anaconda3/envs/my_train/etc/conda/deactivate.d/deactivate.sh also according to the Tutorial
export LD_LIBRARY_PATH=$ORIGINAL_LD_LIBRARY_PATH
unset ORIGINAL_LD_LIBRARY_PATH
However, once I activate the anaconda environment and run my code, I have the following error:
2022-09-19 16:22:13.741682: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-09-19 16:22:15
[INFO] loading dataset...
[INFO] building siamese network...
2022-09-19 16:22:17.738070: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2022-09-19 16:22:17.762785: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.763008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro K620 computeCapability: 5.0
coreClock: 1.124GHz coreCount: 3 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 26.82GiB/s
2022-09-19 16:22:17.763041: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-09-19 16:22:17.764295: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-09-19 16:22:17.765224: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-09-19 16:22:17.765474: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-09-19 16:22:17.766920: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-09-19 16:22:17.767940: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-09-19 16:22:17.769884: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-09-19 16:22:17.770010: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.770274: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.770439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-09-19 16:22:17.770642: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-19 16:22:17.798099: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3600000000 Hz
2022-09-19 16:22:17.798732: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5577de255030 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-09-19 16:22:17.798772: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2022-09-19 16:22:17.825753: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.826097: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5577e01ca4c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-09-19 16:22:17.826117: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro K620, Compute Capability 5.0
2022-09-19 16:22:17.826316: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.826513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro K620 computeCapability: 5.0
coreClock: 1.124GHz coreCount: 3 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 26.82GiB/s
2022-09-19 16:22:17.826542: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-09-19 16:22:17.826603: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-09-19 16:22:17.826627: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-09-19 16:22:17.826664: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-09-19 16:22:17.826705: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-09-19 16:22:17.826758: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-09-19 16:22:17.826799: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-09-19 16:22:17.826875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.827097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-19 16:22:17.827271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-09-19 16:22:17.827304: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
File "train_contrastive_siamese_network.py", line 214, in <module>
run_experiment("rmsprop", pairTrain, labelTrain, pairVal, labelVal)
File "train_contrastive_siamese_network.py", line 103, in run_experiment
featureExtractor = build_siamese_model(config.IMG_SHAPE)
File "/home/ispra-e3/PycharmProjects/my_train/pyimagesearch/siamese_network_mnist.py", line 17, in build_siamese_model
x = Conv2D(64, (2, 2), padding="same", activation="relu", kernel_regularizer=tf.keras.regularizers.l1(l=0.01))(inputs)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 925, in __call__
return self._functional_construction_call(inputs, args, kwargs,
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1098, in _functional_construction_call
self._maybe_build(inputs)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2643, in _maybe_build
self.build(input_shapes) # pylint:disable=not-callable
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/layers/convolutional.py", line 197, in build
self.kernel = self.add_weight(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 597, in add_weight
variable = self._add_variable_with_custom_getter(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 745, in _add_variable_with_custom_getter
new_variable = getter(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 133, in make_variable
return tf_variables.VariableV1(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 260, in __call__
return cls._variable_v1_call(*args, **kwargs)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 206, in _variable_v1_call
return previous_getter(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 199, in <lambda>
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/variable_scope.py", line 2583, in default_variable_creator
return resource_variable_ops.ResourceVariable(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 264, in __call__
return super(VariableMetaclass, cls).__call__(*args, **kwargs)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1507, in __init__
self._init_from_args(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1651, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/keras/initializers/initializers_v2.py", line 397, in __call__
return super(VarianceScaling, self).__call__(shape, dtype=_get_dtype(dtype))
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/init_ops_v2.py", line 561, in __call__
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/init_ops_v2.py", line 1043, in random_uniform
return op(
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/ops/random_ops.py", line 288, in random_uniform
shape = tensor_util.shape_tensor(shape)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/tensor_util.py", line 1029, in shape_tensor
return ops.convert_to_tensor(shape, dtype=dtype, name="shape")
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1499, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 338, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 263, in constant
return _constant_impl(value, dtype, shape, name, verify_shape=False,
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 275, in _constant_impl
return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 300, in _constant_eager_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 97, in convert_to_eager_tensor
ctx.ensure_initialized()
File "/home/ispra-e3/.local/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 539, in ensure_initialized
context_handle = pywrap_tfe.TFE_NewContext(opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
Some final remarks:
(i) if I remove the activate.sh, the script runs. However, it does not use the GPU (there are some warnings about it and I cannot see anything running on nvidia-smi).
(ii) I downloaded cudnn 9.0 files and moved the lib64 and include folders to /usr/local/cuda-9.0. I am not sure if this cuda 9.0 is the right one, but i downloaded it as it contains 'libcudnn.so.7'. Was it the right option?
(iii) it seems the tensorflow knows where libcudnn.so.7 and also the *.so.10 files, but there's still an error. What could that be?
(iv) BTW, I have only tensorflow==2.3.0, I cannot install with conda tensorflow-gpu==2.3.0. Is that a source of the issue?
(v) did I forget something else? how to solve such issue?