Speeding up docker building process for Python project that uses numpy

user_567950 0
Viewed 86

I am building a Docker container using GitHub Actions for one of my Python/Django projects that is also using numpy.

Unfortunately the building process often takes more than 20 minutes, because everytime numpy is being build again.

Also trying to use wheel for it doesn't speed up the process. As I am using versions in my requirements.txt it is also not possible to just install the package using apt.

My Dockerfile looks like this:

FROM python:alpine
EXPOSE 8000

RUN apk update
RUN apk add --no-cache git gcc musl-dev libffi-dev libxml2-dev libxslt-dev gcc swig g++
RUN apk add --no-cache jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev
RUN apk add --no-cache postgresql postgresql-dev
RUN apk add --no-cache bash ffmpeg libmagic
RUN python3 -m pip install --upgrade pip setuptools wheel

RUN mkdir /opt/proj
WORKDIR /opt/proj
COPY . .

RUN pip3 install wheel
RUN pip3 install -r requirements.txt
RUN pip3 install gunicorn
CMD sh -c 'cat /opt/proj/.env'
CMD sh -c 'gunicorn --conf python:proj.gunicorn_conf proj.wsgi --bind 0.0.0.0:8000 --reload --log-level info --access-logfile - --timeout 360 --error-logfile -'

Any suggestions or hints are welcome.

dockerfile github-actions docker pip numpy
1 Answers

I'm not a python developer, and not sure in what step numpty is built, but looking at your Dockerfile, changing the order of some steps may help.

This step copies all files from your project (minus files in .dockerignore it present) into the image:

COPY . .

The build-cache is invalidated if the step above introduced changes, which could be any file in your source code, and which means that any step after this cannot use the build-cache, and will be executed again.

A common pattern is to split installing / building dependencies from building your own code. That way, installing dependencies can be cached, and that cache is not invalidated when changes unrelated to those dependencies are made. (Also see Build your Python image section in the Docker documentation, which is a sample workflow for Python projects)

# run steps that do not depend on requirements.txt nor your project's code
RUN pip3 install wheel
RUN pip3 install gunicorn

# copy requirements.txt and install dependencies defined in it
COPY requirements.txt .
RUN pip3 install -r requirements.txt

# copy your project's code, and build/run your project
COPY . .
CMD sh -c 'cat /opt/proj/.env'
CMD sh -c 'gunicorn --conf python:proj.gunicorn_conf proj.wsgi --bind 0.0.0.0:8000 --reload --log-level info --access-logfile - --timeout 360 --error-logfile -'

Again, I'm not a Python developer, and not familiar with your projects, so some steps in my example may be in the wrong place (but hopefully illustrate the general idea)

user_18115010
Related