I'm trying to run a simple tensorflow example using the tensorflow provided gpu-enabled docker image, from within the Pycharm IDE. Everything works except that when I run it, tensorflow does not detect GPU from the container and defaults back to CPU :
tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit:
Running: Ubuntu 18.04.3 Docker 19.03.3 latest version of NVIDIA docker support as per : https://github.com/NVIDIA/nvidia-docker
I have set up my Pycharm project with the remote interpreter feature to run the image:tensorflow:latest-gpu
If I run the container from the command line with:
docker run --gpus all --rm tensorflow/tensorflow:latest-gpu nvidia-smi
I get this:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:41:00.0 On | N/A |
| 28% 26C P8 9W / 250W | 443MiB / 11177MiB | 1% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
..which tells me that the docker install, the image, and the nvidia docker support are all ok.
Now when Pycharm runs the container, it does NOT include "--gpus all" command line option.
If I run the same command above without the --gpus all parameter:
docker run --rm tensorflow/tensorflow:latest-gpu nvidia-smi
I get:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"nvidia-smi\": executable file not found in $PATH": unknown.
So this would point to Pycharm not adding a flag as the culprit.
However, according to documentation at : https://docs.docker.com/config/containers/resource_constraints/ (bottom section on GPU) the environment variable NVIDIA_VISIBLE_DEVICES=all should accomplish the same thing. I confirmed this environment variable does get set even without the --gpus all param.
Additionally, there does not seem to be a way to add additional command line params from within Pycharm. So I am stuck. I feel that this setup is not too exotic and hope that I am missing something basic.