Issue with Python statsmodels requirement in Docker image build

user_10634126 0
Viewed 1704

I'm trying to build a Docker image for an AWS Batch job, but a new requirement (the Python package statsmodels) is breaking my build process.

requirements.txt lines:

pandas
numpy
statsmodels

Complete Dockerfile:

FROM python:3

COPY requirements.txt /home/
COPY .env /home/
COPY src home/src
COPY process.py /home/

WORKDIR /home/
ENV TZ=America/New_York
RUN /usr/local/bin/python -m pip install --upgrade pip
RUN pip install -r ./requirements.txt

ENV PYTHONPATH='.'

Error log excerpts:

#12 271.7 INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
#12 288.4 INFO: pip is looking at multiple versions of us to determine which version is compatible with other requirements. This could take a while.
#12 288.4 Collecting us
#12 288.4   Downloading us-2.0.1.tar.gz (13 kB)
#12 288.8   Downloading us-2.0.0.tar.gz (13 kB)
#12 289.3   Downloading us-1.0.0.tar.gz (13 kB)
#12 289.8   Downloading us-0.9.1.tar.gz (13 kB)
#12 290.2   Downloading us-0.9.0.tar.gz (13 kB)
#12 290.6   Downloading us-0.8.0.tar.gz (12 kB)
#12 291.2   Downloading us-0.7.1.tar.gz (12 kB)
#12 291.5 INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
#12 291.5 INFO: pip is looking at multiple versions of us to determine which version is compatible with other requirements. This could take a while.
#12 291.6   Downloading us-0.7.tar.gz (11 kB)
#12 291.9   Downloading us-0.6.tar.gz (9.9 kB)
#12 292.3   Downloading us-0.5.tar.gz (9.3 kB)
#12 292.7   Downloading us-0.4.tar.gz (9.3 kB)
#12 293.0   Downloading us-0.3.tar.gz (7.5 kB)
#12 293.4 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
#12 293.4 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
#12 293.4   Downloading us-0.2.tar.gz (6.9 kB)
#12 294.0   Downloading us-0.1.tar.gz (6.3 kB)
#12 294.4 INFO: pip is looking at multiple versions of statsmodels to determine which version is compatible with other requirements. This could take a while.
#12 294.9 Collecting statsmodels
#12 294.9   Downloading statsmodels-0.12.0.tar.gz (17.5 MB)
#12 297.5   Installing build dependencies: started
#12 309.5   Installing build dependencies: finished with status 'done'
#12 309.5   Getting requirements to build wheel: started
#12 392.2   Getting requirements to build wheel: still running...
#12 400.0   Getting requirements to build wheel: finished with status 'done'
#12 400.1     Preparing wheel metadata: started
#12 401.3     Preparing wheel metadata: finished with status 'done'
#12 401.9   Downloading statsmodels-0.11.1.tar.gz (15.4 MB)
#12 404.2   Installing build dependencies: started
#12 415.7   Installing build dependencies: finished with status 'done'
#12 415.7   Getting requirements to build wheel: started
#12 480.8   Getting requirements to build wheel: still running...
#12 481.5   Getting requirements to build wheel: finished with status 'done'
#12 481.5     Preparing wheel metadata: started
#12 482.6     Preparing wheel metadata: finished with status 'done'
#12 483.1   Downloading statsmodels-0.11.0.tar.gz (15.4 MB)
#12 485.5   Installing build dependencies: started
#12 488.6   Installing build dependencies: finished with status 'error'
#12 488.6   ERROR: Command errored out with exit status 1:
#12 488.6    command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_0xxe4fy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.29.14' 'numpy==1.14.5; python_version=='"'"'3.5'"'"'' 'numpy==1.14.5; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"'' 'numpy==1.17.5; python_version>='"'"'3.8'"'"'' 'scipy>=1.0'
#12 488.6        cwd: None
#12 488.6   Complete output (13 lines):
#12 488.6   Ignoring numpy: markers 'python_version == "3.5"' don't match your environment
#12 488.6   Ignoring numpy: markers 'python_version == "3.6"' don't match your environment
#12 488.6   ERROR: Cannot install numpy==1.14.5 and numpy==1.17.5 because these package versions have conflicting dependencies.
#12 488.6   
#12 488.6   The conflict is caused by:
#12 488.6       The user requested numpy==1.14.5
#12 488.6       The user requested numpy==1.17.5
#12 488.6   
#12 488.6   To fix this you could try to:
#12 488.6   1. loosen the range of package versions you've specified
#12 488.6   2. remove package versions to allow pip attempt to solve the dependency conflict
#12 488.6   
#12 488.6   ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
#12 488.6   ----------------------------------------
#12 488.6 ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_0xxe4fy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.29.14' 'numpy==1.14.5; python_version=='"'"'3.5'"'"'' 'numpy==1.14.5; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"'' 'numpy==1.17.5; python_version>='"'"'3.8'"'"'' 'scipy>=1.0' Check the logs for full command output.
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c pip install -r ./requirements.txt]: runc did not terminate sucessfully

Updated Dockerfile per comments:

FROM debian:10-slim

# install Python, pip, and misc build needs
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        build-essential \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean

# bring in program logic
COPY .env /root/.env  # this may need to go into /root/.env (or a custom user)
COPY src /app/
COPY processer.py submit-processer.py /app/
COPY requirements.txt /app/

# install Python dependencies
RUN pip3 install --no-cache-dir -r /app/requirements.txt

# set env vars
ENV TZ=America/New_York

# running process logic
WORKDIR /app
ENTRYPOINT ["python3", "processer.py"]

Error log of updated process:

[+] Building 0.7s (7/12)                                                        
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 979B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/debian:10-slim          0.6s
 => [internal] load build context                                          0.0s
 => => transferring context: 15.91kB                                       0.0s
 => CANCELED [1/8] FROM docker.io/library/debian:10-slim@sha256:062bbd9a1  0.0s
 => => resolve docker.io/library/debian:10-slim@sha256:062bbd9a1a58c9c5b8  0.0s
 => CACHED [2/8] RUN apt-get update     && apt-get install --no-install-r  0.0s
 => ERROR [3/8] COPY .env /root/.env  # this may need to go into /root/.e  0.0s
------
 > [3/8] COPY .env /root/.env  # this may need to go into /root/.env (or a custom user):
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/#" not found: not found
dockerfile statsmodels docker numpy python
1 Answers

It's not entirely clear why from your Dockerfile (I'm surprised the tag 3 isn't sufficient to just pick the latest stable of 3.9.1-buster), but pip is trying to resolve dependencies for several versions of Python (implying there are several installed in the base container)

#12 488.6 ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_0xxe4fy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.29.14' 'numpy==1.14.5; python_version=='"'"'3.5'"'"'' 'numpy==1.14.5; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"'' 'numpy==1.17.5; python_version>='"'"'3.8'"'"'' 'scipy>=1.0' Check the logs for full command output.

Specifying a minor version may clear this up.

If you don't have a particular version need, 3.8-slim-buster (Debian 10 with Python 3.8 .. actually 3.8.9 for now though this may change with bug fixes to that version) may be a practical choice and fix this for you!

from python:3.8-slim-buster
...

If this does not fix the error, then it's likely your 3 tag is just fine and instead extremely-likely that your .env is problematic (presumably it's a Python virtual environment you are trying to preserve).

Regenerate the venv within the context of your container with a new RUN block rather than COPY-ing what your host system has.

After the comments, I think it's worth trying a different Dockerfile which is much more what I would use with at least the following changes

  • base directly off Debian 10 (you may find another container is more efficient/smaller/whatever, but it's extremely likely this works out of the box)
  • install Python 3 yourself
  • don't use /home/ as a base
FROM debian:10-slim

# install Python, pip, and misc build needs
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        build-essential \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean

# bring in program logic
COPY .env /app/  # this may need to go into /root/.env (or a custom user)
COPY src /app/
COPY process.py /app/
COPY requirements.txt /app/

# install Python dependencies
RUN pip3 install --no-cache-dir -r /app/requirements.txt

# set env vars
ENV TZ=America/New_York

# running process logic
WORKDIR /app
ENTRYPOINT ["python3", "process.py"]

If this works

  • using /home may be problematic (rather than /home/someuser/...)
  • there could be something weird with the upstream python base container (I have never personally used it, though that would also be surprising)

If this does not work

  • you may not be working in the directory you think you are (ie. copying the current directory in, another similar-looking directory, etc.)
  • your src may have something bizarre in it
  • statsmodels may be broken (it's versioned 0.12, so that's not totally unreasonable)
user_45410450
Related