How to install local pip package from mounted volume in docker-compose

user_2363197 0
Viewed 568

I am working on a project which has dependency on another project.

In live environment, the dependency is published and installed simply using pip install. On my local environment, I'd like to be able to install the local dependency instead, using the pip install -e command.

The structure is as follow:

- Home
--- Project1
----- docker-compose
----- Dockerfile
--- relaton-py

In this structure, Project1 has dependency on relaton-py, thus I'd like to "install" this dependency using the local relaton-py.

My docker-compose file looks like:

volumes:
      - .:/code
      - /Users/myuser/Dev/Projects/relaton-py:/relaton-py

while the Dockerfile looks like:

COPY requirements.txt /code/requirements.txt

RUN ["pip", "install", "-e", "relaton-py"]

WORKDIR /code

RUN ["pip", "install", "-r", "requirements.txt"]

# Copy the rest of the codebase
COPY . /code

When trying to spin un the environment, I get the following error:

=> ERROR [local/web-precheck:latest 11/19] RUN ["pip", "install", "-e", "relaton-py"]                                                                                                                 1.2s
------
 > [local/web-precheck:latest 11/19] RUN ["pip", "install", "-e", "relaton-py"]:
#27 0.685 ERROR: relaton-py is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
#27 0.889 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
#27 0.889 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
failed to solve: rpc error: code = Unknown desc = executor failed running [pip install -e relaton-py]: exit code: 1

However, if I try not to install this local dependency in the Dockerfile, the entire environment spins up and I am able to access the container and install the package manually with the same command pip install -e relaton-py.

Has anyone had to deal with this sort of thing already? Any idea on how to make Dockerfile recognise the files in the mounted volumes?

dockerfile docker-compose docker pip python
2 Answers

The volume is only bound when you run the container, the pip install -e relaton-py instead is happening when you build the image so the volume bind has not happened yet.

I would suggest embedding relaton-py inside the container.

To do so you need to change the build context to include the relaton-py directory. So in docker-compose.yml

services:
  Project1:
    build:
      context: ../
      dockerfile: ./Project1/Dockerfile

And andjust your Dockerfile accordingly

COPY ./relaton-py /relaton-py
RUN ["pip", "install", "-e", "/relaton-py"]

WORKDIR /code

COPY ./Project1/requirements.txt requirements.txt
RUN ["pip", "install", "-r", "requirements.txt"]

COPY ./Project1 .

This way the code will be available when you build the image.

Then if you need your local code changes to happen in the container as well you can mount your local directory on top of the directory inside the container by doing:

volumes:
      - .:/code
      - ../relaton-py:/relaton-py
user_133847740

For simpler packages, it may be enough to just mount it directly into the site-packages.

In your example, you might want to try:

  1. putting relaton-py (just the package in its source form, e.g. the Git repository) one directory above your Compose file, and
  2. adding this line to docker-compose.yml under volumes:
- ../relaton-py/relaton:/usr/local/lib/python3.10/site-packages/relaton:ro

Assuming you use Python 3.10, this should override whatever gets installed from PyPI during Docker image build with the directory on your host system, mounted in read-only mode to prevent accidental changes from within the container (but any source code changes you make on your main system, of course, will be propagated immediately).

If your package includes native modules and requires compilation for particular OS/architecture, there may be extra steps involved in this.

user_2474410
Related