How to use setup.py and setup.cfg and pip-tools to obtain layered requirements.txt under different environment of a Django project?

user_80353 0
Viewed 1081

I'm using https://github.com/jazzband/pip-tools to handle compiling the requirements.txt for a Django project.

Previously, I was using without a setup.py and so i was using base.in, local.in, and production.in.

When I needed a local requirements.txt after i finish pip-compile, I just run pip-sync base.txt local.txt and it will install the requirements for local environment.

When I needed a production requirements.txt after i finish pip-compile, I just run pip-sync base.txt production.txt and it will install the requirements for production environment.

So I switch away from using base.in is because I wanted to also lock the python version and i realized, setup.py and setup.cfg can help using python_requires

But now i become unsure of how to use setup.py and setup.cfg along with pip-tools to compile requirements.txt that can be environment-specific.

The only documentation for layered requirements is by using the different .in files as written in the README as https://github.com/jazzband/pip-tools#workflow-for-layered-requirements

So my question is:

Given:

  1. pip-tools
  2. setup.py and setup.cfg

how to still have layered requirements?

pip-tools pip python
2 Answers

pip-tools works smoothly with setup.py too. You just need to run it without providing it with an *.in file.

Mixed variant with setup.py and in files:

So assuming, that you have the following structure:

# setup.py replacing the base.in file
from distutils.core import setup


setup(
    name="MyLibrary",
    version="1.0",
    install_requires=[
        "requests",
        "bcrypt",
    ]
)

and a local.in file:

django

you need to do the following to compile the dependencies:

$ pip-compile -o base.txt 
$ pip-compile local.in
$ pip-sync base.txt local.txt

$ pip-compile -o base.txt will generate dependencies from setup.py using base.txt as an output file. It defaults to requirements.txt. $ pip-compile local.in is the same what you did before, as with the pip-sync part which doesn't change.

So the only magic here is to run pip-compile without providing it with an input file.

Setup.py only solution:

Setup.py supports extras_require which is a dictionary of named optional dependencies:

[...]
    extras_require={
        "local": ["pytest"],
    },
[...]

pip-tools has an option extra:

$ pip-compile --help |grep extra
  --extra TEXT     Name of an extras_require group to install;

So you could do the following:

pip-compile --extra local -o local.txt
pip-compile --extra production -o production.txt

The output files contain all requiremenets in install_requires + the extra requirements specified.

Afterwards you just sync the local/production.txt:

$ pip-sync local.txt
$ pip-sync production.txt

If I was you, I would grab the pure setup.py variant.

user_140337650

Can you check pipenv. pipenv uses Pipfile and Pipfile.lock for dependencies instead of requirements.txt. It has a clear separation between the dependencies you install and the dependencies of the dependencies.

check this sample below it is much clearer:

╰─ pipenv graph                                                                                                                                                  ─╯
boto3==1.20.38
  - botocore [required: >=1.23.38,<1.24.0, installed: 1.23.38]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
    - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
      - six [required: >=1.5, installed: 1.16.0]
    - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.8]
  - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
  - s3transfer [required: >=0.5.0,<0.6.0, installed: 0.5.0]
    - botocore [required: >=1.12.36,<2.0a.0, installed: 1.23.38]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
        - six [required: >=1.5, installed: 1.16.0]
      - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.8]
colorama==0.4.4
pylint==2.12.2
  - astroid [required: >=2.9.0,<2.10, installed: 2.9.3]
    - lazy-object-proxy [required: >=1.4.0, installed: 1.7.1]
    - setuptools [required: >=20.0, installed: 58.1.0]
    - typing-extensions [required: >=3.10, installed: 4.0.1]
    - wrapt [required: >=1.11,<1.14, installed: 1.13.3]
  - isort [required: >=4.2.5,<6, installed: 5.10.1]
  - mccabe [required: >=0.6,<0.7, installed: 0.6.1]
  - platformdirs [required: >=2.2.0, installed: 2.4.1]
  - toml [required: >=0.9.2, installed: 0.10.2]
  - typing-extensions [required: >=3.10.0, installed: 4.0.1]
pytest-cov==3.0.0
  - coverage [required: >=5.2.1, installed: 6.0.2]
  - pytest [required: >=4.6, installed: 6.2.5]
    - attrs [required: >=19.2.0, installed: 21.2.0]
    - iniconfig [required: Any, installed: 1.1.1]
    - packaging [required: Any, installed: 21.0]
      - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - pluggy [required: >=0.12,<2.0, installed: 1.0.0]
    - py [required: >=1.8.2, installed: 1.10.0]
    - toml [required: Any, installed: 0.10.2]
PyYAML==6.0
requests==2.27.1
  - certifi [required: >=2017.4.17, installed: 2021.10.8]
  - charset-normalizer [required: ~=2.0.0, installed: 2.0.10]
  - idna [required: >=2.5,<4, installed: 3.3]
  - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.8]
tomli==1.2.1

It has also support for dev and prod package definitions like this:

╰─ cat Pipfile                                                                                                                                                   ─╯
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
colorama = "*"
boto3 = "*"
pyyaml = "*"
requests = "*"
pylint = "*"

[dev-packages]
pytest = "*"
pytest-cov = "*"

[requires]
python_version = "3.9"

For more details please check also this link: https://realpython.com/pipenv-guide/

Also check this for multiple environments in pipenv: Pipenv: Multiple Environments

user_90410870
Related