building docker: opt/conda/bin/conda not found

user_14086325 0
Viewed 2050

I am completely new to docker, and I need to run a code in docker environment.

I get error in building Dockerfile :

I'm running Ubuntu 20.04 via hyper-V, and when I build the Dockerfile, I get the following message:

Step 4/20 : RUN curl -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  &&      chmod +x ~/miniconda.sh &&      ~/miniconda.sh -b -p /opt/conda &&      rm ~/miniconda.sh &&      /opt/conda/bin/conda install numpy pyyaml scipy ipython mkl &&      /opt/conda/bin/conda install -c soumith magma-cuda90 &&      /opt/conda/bin/conda clean -ya <br />
 ---> Running in 9758f4fe60a4 <br />
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 

                                 Dload  Upload   Total   Spent    Left  Speed 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 
/bin/sh: 1: /opt/conda/bin/conda: not found
The command '/bin/sh -c curl -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  &&      chmod +x ~/miniconda.sh &&      ~/miniconda.sh -b -p /opt/conda &&      rm ~/miniconda.sh &&      /opt/conda/bin/conda install numpy pyyaml scipy ipython mkl &&      /opt/conda/bin/conda install -c soumith magma-cuda90 &&      /opt/conda/bin/conda clean -ya' returned a non-zero code: 127

Dockerfile :

# PyTorch Install
FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

RUN apt-get update && apt-get install -y --no-install-recommends \
         build-essential \
         cmake \
         git \
         curl \
         vim \
     emacs \
         parallel \
         ca-certificates \
         libjpeg-dev \
     hdf5-tools \
         libpng-dev &&\
     rm -rf /var/lib/apt/lists/*


RUN mkdir ~/.parallel && touch ~/.parallel/will-cite

RUN sudo curl -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
     chmod +x ~/miniconda.sh && \
     ~/miniconda.sh -b -p /opt/conda && \     
     rm ~/miniconda.sh && \
     /opt/conda/bin/conda install numpy pyyaml scipy ipython mkl && \
     /opt/conda/bin/conda install -c soumith magma-cuda90 && \
     /opt/conda/bin/conda clean -ya 
ENV PATH /opt/conda/bin:$PATH 
# This must be done before pip so that requirements.txt is available
WORKDIR /opt/pytorch
COPY . .
RUN conda install pytorch torchvision -c pytorch

#
# Now install the julia dependencies.
#
WORKDIR /opt/julia
RUN pip install pandas matplotlib utils argh biopython
RUN conda install networkx joblib

RUN apt-get update && apt-get install -y curl
RUN mkdir /julia
RUN curl -L https://julialang-s3.julialang.org/bin/linux/x64/0.6/julia-0.6.2-linux-x86_64.tar.gz | tar -C /julia --strip-components=1  -xzf -
ENV PATH "/julia/bin:$PATH"
RUN julia -e "Pkg.init()"
COPY setup.jl /julia/setup.jl
RUN julia /julia/setup.jl

WORKDIR /root/hyperbolics
ENV PYTHONPATH /root/hyperbolics

when I tried commands under RUN directly at command prompt, it works well.

However, every command is 'not found' when I build Dockerfile, julia, pip, etc.. (when I comment out the conda... part)
How can I solve this problem?

docker linux
1 Answers

The /bin/sh: 1: /opt/conda/bin/conda: not found error is caused by conda not being installed properly. This is because the miniconda.sh file that you download (with curl -o ~/miniconda.sh -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh) is an empty file.

This happens because curl by default doesn't follow redirects, and the response from the url above sends a redirect rather than the miniconda.sh file directly. You can verify this by checking the headers of the URL (e.g. curl -i https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh shows a HTTP 301 redirect statuscode).

You can fix this by telling curl to follow redirects, by providing the -L flag in your curl command, e.g. :

(I also had to remove sudo)

RUN curl -L -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
     chmod +x ~/miniconda.sh && \
     ~/miniconda.sh -b -p /opt/conda && \     
     rm ~/miniconda.sh && \
     /opt/conda/bin/conda install numpy pyyaml scipy ipython mkl && \
     /opt/conda/bin/conda install -c soumith magma-cuda90 && \
     /opt/conda/bin/conda clean -ya

Here is some debugging info/research:

You can verify the blank miniconda.sh is the problem by looking at the output of curl in your snippet above. It looks a bit cryptic, but you can see nothing has actually been downloaded

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 

                                 Dload  Upload   Total   Spent    Left  Speed 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

When I tried building the docker image with the -L flag added, the output looked like this:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
100 88.7M  100 88.7M    0     0  11.9M      0  0:00:07  0:00:07 --:--:-- 34.7M

You can also manually check the contents of miniconda.sh by deleting everything up to the download/run part

(e.g. keeping everything up to this part)

RUN sudo curl -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
     chmod +x ~/miniconda.sh && \
     ~/miniconda.sh -b -p /opt/conda

and then running sudo docker run -it containername /bin/bash and checking the contents of the miniconda.sh (cat ~/miniconda.sh, which reveals the file is empty).

user_139857650
Related