getting no develop install with `pip install -e .` unless I delete `pyproject.toml`

user_9498105 0
Viewed 164

I have the following situation that pip install -e . does not build the develop version unless I delete the pyproject.toml which does not contain packaging information, but black configuration.

Does somebody know what to do in order to get the develop build.

my pyproject.toml looks like this:

# Source https://github.com/psf/black#configuration-format
[tool.black]
line-length = 100
target-version = ['py38']
exclude = '''

'''

setup.py

from setuptools import find_namespace_packages
from setuptools import setup

setup(
    name="webservice",
    packages=find_packages(),
    version="0.1.0",
    description="description",
    author="Author",
    license="License",
)

running pip install -e . with these two files...

(webservice_tool)pk@LAP1:~/webservice_tool$ pip install -e .
Obtaining file:///home/pk/webservice_tool
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: webservice
  Building editable for webservice (pyproject.toml) ... done
  Created wheel for webservice: filename=webservice-0.1.0-0.editable-py3-none-any.whl size=4070 sha256=dcb7c034ba437503d1059fe9370ccafbda144cd19f3e5d92340a63a7da396422
  Stored in directory: /tmp/pip-ephem-wheel-cache-6iqiqbob/wheels/e6/b5/ba/40d8c3d66df94ee2ae46e181034e0c3c47132784db53284d0b
Successfully built webservice
Installing collected packages: webservice
Successfully installed webservice-0.1.0

I delete pyproject.toml and only then Running setup.py develop shows up.

(webservice_tool) pk@LAP1:~/webservice_tool$ pip install -e .
Obtaining file:///home/pk/webservice_tool
  Preparing metadata (setup.py) ... done
Installing collected packages: webservice
  Attempting uninstall: webservice
    Found existing installation: webservice 0.1.0
    Uninstalling webservice-0.1.0:
      Successfully uninstalled webservice-0.1.0
  Running setup.py develop for webservice
Successfully installed webservice-0.1.0

versions of some selected packages from my conda env, running within wsl2

packaging                 21.3               pyhd3eb1b0_0  
pip                       22.1.2           py38h06a4308_0  
python                    3.8.13               h12debd9_0  
setuptools                61.2.0           py38h06a4308_0

folder structure

|-- data_utils
|   |-- clean_cache.py
|   `-- advanced_utils.py
|-- deployment
|   |-- base_deployment
|   |   |-- auth-proxy.yaml
|   |   |-- kustomization.yaml
|   |   |-- webapi.yaml
|   |   `-- webui.yaml
|   `-- mysql_from_helm
|       |-- mysql-from-helm.yaml
|       `-- mysql-kustomization.yaml
|-- docker-compose.yml
|-- Dockerfile
|-- environment.yml
|-- live_api
|   |-- definitions.json
|   |-- __init__.py
|   `-- live_api.py
|-- params.py
|-- pyproject.toml
|-- setup.py
|-- shared_helpers
|   |-- data_cleaning.py
|   |-- handle_time.py
|   |-- __init__.py
|   |-- plot_timesequence.py
|   |-- read_samples.py
|   `-- save_samples.py
|-- setup.py
|-- util.py
|-- webtool
|   |-- clean_data
|   |   |-- clean_data.py
|   |   `-- __init__.py
|   |-- evaluation
|   |   |-- draw_figures.py
|   |   |-- __init__.py
|   |   `-- webtool_metrics.py
|   |-- __init__.py
|   |-- preprocess
|   |   |-- __init__.py
|   |   `-- preprocess.py
|   |-- ui
|   |   |-- __init__.py
|   |   `-- create_ui.py
|   `-- util
|       |-- data_input.py
|       |-- data_redefinitions.py
|       `-- __init__.py
|-- webtool.egg-info
|   |-- dependency_links.txt
|   |-- entry_points.txt
|   |-- PKG-INFO
|   |-- SOURCES.txt
|   `-- top_level.txt
`-- webtool_tests
    |-- clean_data
    |   `-- test_clean_data.py
    |-- evaluation
    |   `-- test_draw_figures.py
    |-- preprocess
    |   `-- test_preprocess.py
    `-- util
        |-- test_data_input.py
        `-- test_data_redefinitions.py
pyproject.toml pip setup.py setuptools python
2 Answers

These are both development installs. The difference in the pip output here is because the presence (or absence) of a pyproject.toml file causes pip to use the build backend hooks (or not). From PEP 517:

If the pyproject.toml file is absent ... the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py

You can also control that with a pip command line option:

$ pip install --help | grep pep
  --use-pep517                Use PEP 517 for building source distributions
                              (use --no-use-pep517 to force legacy behaviour).

The difference is that with a PEP 517 style build, pip is setting up a venv and freshly installing setuptools for the purposes of the package build behind the scenes (see "Installing build dependencies ... done" in the log), versus invoking python setup.py develop directly where it is just assumed that an adequate setuptools version is already installed in the Python runtime which you used to execute the setup.py. The point here is that using a PEP 517 style build system allows the project to specify the setuptools version it requires (or, indeed, to use some other build system entirely).

The end result will be the same - a .pth path configuration file placed in site-packages will expose the source directory as a development installation.

Since util.py is not contained in any package, for it to be picked up in the development installation (as opposed to just importing from the current working directory), you'll also need to list it alongside find_packages() in the setup call:

# in setup.py 
from setuptools import setup

setup(
    name="webservice",
    version="0.1.0",
    packages=find_packages(),
    py_modules=["util"],  # <--- here
    ...
)
user_6740390

As others have already said, both of these installs are editable. The difference in import behavior is caused by the way an editable install is performed under the hood:

According to your setup.py, you don't specify any modules in a project. You need to list them in py_modules argument of setup() function.
You could also move all your project con­figu­ration into pyproject.toml file.

user_117257530
Related