When I try to run a very minimal Tensorflow example:
import tensorflow as tf
c = tf.constant([1,2,3])
The system hangs forever (at least for ten minutes) with no sign of what it is doing. It uses 100% of one virtual CPU core when in this state. When run in a Juypter notebook the kernel outputs this to the console:
2020-03-31 11:12:04.840507: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840576: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840589: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-03-31 11:12:05.521172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-31 11:12:05.539193: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.539639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7845GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-03-31 11:12:05.539841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-03-31 11:12:05.541113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-03-31 11:12:05.542119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-03-31 11:12:05.542324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-03-31 11:12:05.543632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-03-31 11:12:05.544401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-03-31 11:12:05.547212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-31 11:12:05.547337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-03-31 11:12:05.567845: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3393550000 Hz
2020-03-31 11:12:05.568364: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564107e16440 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-31 11:12:05.568395: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
I did have Tensorflow working previously on this system, so I think this might be some sort of library issue that got caused by a system update.
The GPU is an Nvidia GTX 1070. Tensorflow version is 2.1.0, and hasn't changed since when it was working. Running Arch Linux, if that matters.
I tried downgrading from CUDA 10.2 to 10.1, but the issue still occurs.
I can also reproduce this with PyTorch:
import torch
import transformers
t = torch.tensor([1,2,3])
t.cuda()
(import transformers prevents a "CUDA: Out of memory" issue - there must be something it does that initializes PyTorch that I don't know how to do.)
This has the same issue, where it freezes pegging one CPU core, though it produces less output:
020-03-31 11:13:41.428483: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428571: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428587: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
I'm pretty sure the complaints about TensorRT are not relevant, because when I had this working previously it would output those as well.
How can I resolve this issue? Or at least, what else can I do to determine what it is doing while frozen?