With multi-stage Dockerfile, the pip wheel still needs the dependencies from the base builder

Viewed 1622

I am new to the Docker environment. So, sorry if asking this is a common thing.

I'm using the Dockerfile below, which first uses a builder to make things with the wheel. The problem is that the cryptography package in my requirements.txt file (a pip package) needs gcc and some other packages to build. As you can see below, I installed these packages (gcc and others) with RUN apk add in the base builder and then built the wheel. However, in the final builder, I get error saying that "gcc: No such file ..." so meaning that the gcc was not installed. (But I'm using the generated wheel from the base builder! Why does it still need the gcc and other stuff?)

If I install the gcc and other build dependencies in the final build too, again, it solves the problem, yes. But, in this case, I lose the benefits of using multi-stage Dockerfile.

I can't understand the problem. Can you help me?

Here's the Dockerfile:

FROM python:3.9.1-alpine as builder

WORKDIR /usr/src/MYAPP

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt .

RUN apk update \
    && apk add --virtual build-deps gcc python3-dev musl-dev \
        libressl-dev libffi-dev cargo \
    && apk add postgresql-dev \
    && apk add jpeg-dev zlib-dev libjpeg \
    && pip install --upgrade pip \
    && pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/MYAPP/wheels -r requirements.txt \
    && apk del build-deps

RUN pip install flake8
COPY . .
RUN flake8 --ignore=E501,F401 .


############
############
### FINAL ##
############
############
FROM python:3.9.1-alpine

RUN mkdir -p /home/MYAPP

RUN addgroup -S MYAPP && adduser -S MYAPP -G MYAPP

# create the appropriate directories
ENV HOME=/home/MYAPP
ENV APP_HOME=/home/MYAPP/web
RUN mkdir $APP_HOME
WORKDIR $APP_HOME

# install dependencies
RUN apk add libpq
COPY --from=builder /usr/src/MYAPP/wheels /wheels
COPY --from=builder /usr/src/MYAPP/requirements.txt .
RUN pip install --no-cache /wheels/*

COPY . $APP_HOME

RUN chown -R MYAPP:MYAPP $APP_HOME

USER MYAPP
1 Answers

OK, after spending hours, I found the problem and fixed it. So I'll answer my own question.

First of all, I was using --no-deps parameter in the pip wheel command. This parameter causes that the pip wheel only downloads the main packages, without their dependencies. So in the second build image (stage), pip was trying to download all the sub dependencies for these main dependencies. So these dependencies were in need for some system packages to build.

Removing this --no-deps parameter would normally fix the problem in such situation, but it caused another error for my situation. There were some version conflicts between the dependencies, which were not exist with regular pip install -r ...

Instead of the pip wheel technique, I just used pip install --user -r option to install my requirements in the folder /root/.local. Then I moved the /root/.local folder to the second stage's /root:

..
COPY --from=builder /root/.local /home/.local
..

That's all. All my requirements were installed in the second build image too.

Do not forget to add the /root/.local/bin path to your $PATH:

ENV PATH=/root/.local/bin:$PATH
Related