I'm running the tensorflow/tensoflow:latest-gpu docker container. I can run simple vector operations like @ for matrix multiplication without a problem. However, when I run the following minimal example:
import tensorflow as tf
tf.linalg.inv(tf.eye(10))
I get the following error:
2021-02-15 16:18:20.375254: I tensorflow/core/util/cuda_solvers.cc:180] Creating CudaSolver handles for stream 0x528cf90
2021-02-15 16:18:20.375365: I tensorflow/stream_executor/platform/default/dso_loader.cc:49]
Successfully opened dynamic library libcusolver.so.10
2021-02-15 16:18:21.854945: I tensorflow/stream_executor/platform/default/dso_loader.cc:49]
Successfully opened dynamic library libcublas.so.11
2021-02-15 16:18:21.934489: F tensorflow/core/util/cuda_solvers.cc:120]
Check failed: cublasCreate(&cublas_handle) == CUBLAS_STATUS_SUCCESS Failed to create cuBlas instance.
The Python interpreter crashes. The CUDA and cuBLAS libraries are successfully opened, so I'm not sure what's causing the error.
The crash also happens with the tensorflow/tensorflow:devel-gpu image. When I try an earlier TensorFlow version (2.3), I do not get the error. However, I need to use >=2.4 because that's required by tensorflow_probability.
I'm on Pop! OS (Ubuntu 20.10), using a GTX 1650.
Edit: Installing tf-nightly natively on the host system doesn't produce the error; tf.linalg.inv(tf.eye(10)) works fine. This does not solve the problem with the docker image (nightly image still produces the error), but I have a working GPU tensorflow environment for now.