How can setup.py reference other files in the source repo?

user_453851 0
Viewed 75

I'm having trouble getting a python package to build with python -m build .. setup.py fails on:

FileNotFoundError: [Errno 2] No such file or directory: 'requirements/requirements.txt'

It's caused by the fact that build is copying files to a temporary directory first. But it's only copying the source/, README.md, setup.py, setup.cfg. It's not copying requirements/.

For complex reasons my setup.py needs to reference other files at the root of the source repo - a directory containing multiple requirements.txt files. It's not really worth discussing why it needs to be structured this way, I've already been through that long debate with colleagues.

This works fine when we install the package through pip install -e . or as a git dependency git+ssh://... but fails when building before we push to a pypi repo.

setup.cfg
setup.py
source/
source/my_package/
requirements/
requirements/requirements.txt
requirements/some-other-requirements.txt

setup.py references this directory before calling setup().

from pathlib import Path
from setuptools import setup, find_namespace_packages


requirements_dir = Path("requirements")


# This is the line that fails:
with (requirements_dir / "requirements.txt").open() as f:
    install_requires = list(f)


setup(
    packages=find_namespace_packages(where="source", include=["acme_corp.*"], exclude=["tests", "tests.*"]),
    package_dir={"": "source"},
    install_requires=install_requires,
    extras_require=optional_packages,
)
setuptools python
1 Answers

build does the correct job here. After building the source dist, it checks whether the built result can be actually installed. And without including the files under requirements into the source dist, the source dist can not be installed and is thus unusable. Try it out:

$ python -m build --sdist  # builds fine, but the tar is broken:
$ cd dist
$ tar tf pkg-0.0.1.tar.gz | grep requirements  # will be empty
$ pip wheel pkg-0.0.1.tar.gz  # will fail
Processing ./pkg-0.0.1.tar.gz
  File was already downloaded 
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command:
   ...
  Complete output (9 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-wfzz27_k/setup.py", line 9, in <module>
      with (requirements_dir / "requirements.txt").open() as f:
    File "/usr/lib64/python3.9/pathlib.py", line 1252, in open
      return io.open(self, mode, buffering, encoding, errors, newline,
    File "/usr/lib64/python3.9/pathlib.py", line 1120, in _opener
      return self._accessor.open(self, flags, mode)
  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-wfzz27_k/requirements/requirements.txt'

To fix, write a MANIFEST.in alongside the setup.py that includes requirements directory in the source dist. Example file contents:

graft requirements

Building a source dist should work now; you can additionally verify that the sdist now contains all files to install from:

$ tar tf dist/pkg-0.0.1.tar.gz | grep requirements
pkg-0.0.1/requirements/
pkg-0.0.1/requirements/requirements.txt
...

This only concerns the source dist, as the wheel is built on host (your machine) already and does not contain a setup script anymore. Wheel building will also ignore the MANIFEST.in file.

user_26502490
Related