Setup.py and Docker layering for building python applications

Question

tl;dr How do I use dockers layering technique without a requirements.txt for python3?

Long story:
A well formed Dockerfile for python should look as follows:

FROM python
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
ENTRYPOINT["python3","sample.py"]

Which increases build times of docker containers significantly because downloading and installing libraries in subsequent builds can be cached by docker.

Now I have a rather complex python application, my team decided to get rid of the requirements.txt and have a single point of truth for our application: our setup.py file.

Which is great from a usability perspective. pip install . for our average user. pip install .[dev] for our developers. If changes happen they happen in our setup.py.

However from a DevOps perspective this is not as nice. Unfortunately it is not possible to execute setup.py without /src being present. But if /src is present before installing our app it is not possible to make use of docker's layering technique.

So it is a trade off between usability and build speeds.

My workaround is to auto generate a requirements.txt for CI/CD purposes which feels odd. However it is still better to auto generate than to manage dependencies in two files (requirements.txt/setup.py).

I try to reduce build times of my containers, while at the same time keeping requirements for developers and end users as comprehensible as possible.

What did I miss?

Here is some code:

# Directory layout
.
├── Dockerfile
├── gen-requirements.py
├── setup.py
└── src
    └── greatapp
        ├── __init__.py
        └── sample.py

Not working Dockerfile

# DOES NOT WORK
FROM python
RUN mkdir -p /home/app/
WORKDIR /home/app/
COPY setup.py setup.py
RUN pip install .[dev]
COPY src/ src/
ENTRYPOINT ["great-app"]

Working Dockerfile

FROM python
RUN mkdir -p /home/app/
WORKDIR /home/app/
COPY ["setup.py", "gen-requirements.py", "./"]
RUN python3 gen-requirements.py \
    && pip install -r requirements.txt 
COPY src/ src/
RUN pip install .[dev]
ENTRYPOINT ["great-app"]

gen-requirements.py

#!/usr/bin/env python3
# Adapted from https://stackoverflow.com/questions/24236266/how-to-extract-dependencies-information-from-a-setup-py
# https://stackoverflow.com/users/748858/mgilson
#
# Gen-requirements extracts requirements from the setup.py to generate a requirements.txt.
# This allows for a single source of dependencies (truth) in our setup.py file.
# Only used for CI purposes.

from unittest import mock
import setuptools

with mock.patch.object(setuptools, "setup") as mock_setup:
    import setup  # This is setup.py which calls setuptools.setup

_, kwargs = mock_setup.call_args

install_requires = kwargs.get("install_requires", [])
extras_require = kwargs.get("extras_require", [])["dev"]

with open("requirements.txt", "w") as f:
    f.write(f"# Auto generated by {__file__} from setup.py.
# Change requirements in setup.py.
")
    f.write("
".join(install_requires + extras_require))

setup.py

#!/usr/bin/env python3
import os
from setuptools import find_packages, setup


install_requirements = [
    "tensorflow",
]

development_requirements = [
    # Tests
    "tox",
]

extras = {"dev": development_requirements}

setup(
    name="greatapp",
    package_dir={"": "src"},
    packages=find_packages(where="src"),
    entry_points="""
        [console_scripts]
        great-app=greatapp.sample:main
    """,
    install_requires=install_requirements,
    extras_require=extras,
)

sample.py

#!/usr/bin/env python3
def main():
    a = 3
    b = 3
    print(a + b)

Setup.py and Docker layering for building python applications

0 Answers