Docker GPU enabled version (19.03) does not load tensorflow successfully

Question

I want to use docker 19.03 and above in order to have GPU support. I currently have docker 19.03.12 in my system. I can run this command to check that Nvidia drivers are running:

docker run -it --rm --gpus all ubuntu nvidia-smi
Wed Jul  1 14:25:55 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 107...  Off  | 00000000:01:00.0 Off |                  N/A |
| 26%   54C    P5    13W / 180W |    734MiB /  8119MiB |     39%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Also, if run locally my module works with GPU support just fine. But if I build a docker image and try to run it I get a message:

ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

I am using cuda 9.0 with tensorflow 1.12.0 but I can switch to cuda 10.0 with tensorflow 1.15.
As I get it the problem is that I am probably using a previous dockerfile version with commands which does not make it compatible with new docker GPU enabled version (19.03 and above).
The actual commands are these:

FROM nvidia/cuda:9.0-base-ubuntu16.04

# Pick up some TF dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        cuda-command-line-tools-9-0 \
        cuda-cublas-9-0 \
        cuda-cufft-9-0 \
        cuda-curand-9-0 \
        cuda-cusolver-9-0 \
        cuda-cusparse-9-0 \
        libcudnn7=7.0.5.15-1+cuda9.0 \
        libnccl2=2.2.13-1+cuda9.0 \
        libfreetype6-dev \
        libhdf5-serial-dev \
        libpng12-dev \
        libzmq3-dev \
        pkg-config \
        software-properties-common \
        unzip \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update && \
        apt-get install nvinfer-runtime-trt-repo-ubuntu1604-4.0.1-ga-cuda9.0 && \
        apt-get update && \
        apt-get install libnvinfer4=4.1.2-1+cuda9.0

I could not find a docker base file for fundamental GPU usage either.

In this answer there was a proposal for exposing libcuda.so.1 but it did not work in my case.

So, is there any workaround for this problem or a base dockerfile to adjust to?

My system is Ubuntu 16.04.

Edit:

I just noticed that nvidia-smi from within docker does not display any cuda version:

CUDA Version: N/A

in contrast with the one locally run. So, this probably means that no cuda is loaded inside docker for some reason I guess.

user_3584765 · Answer

tldr;

A base Dockerfile which seems to work with docker 19.03+ & cuda 10 is this:

FROM nvidia/cuda:10.0-base

which can be conbined with tf 1.14 but for some reason could not found tf 1.15.

I just used this Dockerfile to test it:

FROM nvidia/cuda:10.0-base
CMD nvidia-smi

longer answer:

Well, after a lot of trials and errors (and frustration) I managed to make it work for docker 19.03.12+cuda 10 (although with tf 1.14 not 1.15).

I used the code from this post and used the base Dockerfiles provided there.

First I tried to check the nvidia-smi from within docker using Dockerfile:

FROM nvidia/cuda:10.0-base
CMD nvidia-smi

$docker build -t gpu_test .
...
$docker run -it --gpus all gpu_test
Fri Jul  3 07:31:05 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 107...  Off  | 00000000:01:00.0 Off |                  N/A |
| 45%   65C    P2   142W / 180W |   8051MiB /  8119MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

which finally seems to find cuda binaries: CUDA Version: 10.1.

Then, I made a minimal Dockerfile which could test the successful loading of tensorflow binary libraries within docker:

FROM nvidia/cuda:10.0-base

# The following are just declaring variables and ultimately use
ARG USE_PYTHON_3_NOT_2=True
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}

RUN apt-get update && apt-get install -y \
    ${PYTHON} \
    ${PYTHON}-pip

RUN ${PIP} install tensorflow_gpu==1.14.0

COPY bashrc /etc/bash.bashrc
RUN chmod a+rwx /etc/bash.bashrc

WORKDIR /src
COPY *.py /src/

ENTRYPOINT ["python3", "tf_minimal.py"]

and tf_minimal.py was simply:

import tensorflow as tf

print(tf.__version__)

and for completeness I just post the bashrc file I am using:

# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ==============================================================================

export PS1="$$\e[31m$$tf-docker$$\e[m$$ $$\e[33m$$\w$$\e[m$$ > "
export TERM=xterm-256color
alias grep="grep --color=auto"
alias ls="ls --color=auto"

echo -e "\e[1;31m"
cat<

Docker GPU enabled version (>19.03) does not load tensorflow successfully

1 Answers