Hierarchical python virtualenvs

user_904050 0
Viewed 91

Is it possible to create a hierarchy of python virtualenvs? For example, on an HPC system, imagine that sysadmins install the global python, and library developers create a python virtual environment venv1, and app developers create their own virtual environment venv2.

Can the app developers source venv1, then source venv2 and then install their preferred dependencies without root access or alteration of the library virtualenv venv1?

Naively, I run:

~/tmp ❯❯❯ python3.9 -m venv venv1/
~/tmp ❯❯❯ python3.9 -m venv venv2/
~/tmp ❯❯❯ source venv1/bin/activate.fish
(venv1) ~/tmp ❯❯❯ echo $PATH
~/tmp/venv1/bin /usr/local/bin
(venv1) ~/tmp ❯❯❯ source venv2/bin/activate.fish
(venv2) ~/tmp ❯❯❯ echo $PATH
~/venv2/bin /usr/local/bin/

which doesn't work. So certainly they cannot be nested without some flags to the environments.

virtualenv python-3.x
2 Answers

I had a similar question, and ended up solving it with .pth files (which is similar to how development packages work under the hood).

In my case I have a singularity container which requires a specific version of ROCm on the host, so I wanted to be able to fix/make common the tensorflow-rocm and pytorch versions aligned with the ROCm-dkms on the host. So I created a venv in my home folder, installed pytorch, then made my project folder and put another venv in there, and then added /home/logan/venv/lib/python3.10/site-packages/ to a new common.pth file which I put in /home/logan/project1/venv/lib/python3.10/site-packages/common.pth.

So in other words, it looks like this:

/home/logan
├── venv
│   └── lib/python3.10/site-packages* (pytorch with ROCm installed here)
└── project1
    └── venv
        └── lib/python3.10/site-packages (project1 pkgs installed here)
            └── common.pth (links to * above)

Now it uses the pytorch from the /home/logan/venv but installs it's own deps into /home/logan/project/venv. This seems to work fine for me, hope it helps someone else! And if there are any better ways I'd be open to hearing about them :)

user_21904110

Once you have a virtualenv set up, you can use pip to install any packages you like in it. So there's no advantage to sourcing other environments 'directly'.

You can however generate a requirements.txt listing the packages installed in an existing environment, then use that to then install them in a new one:

venv1/bin/pip freeze >requirements-venv1.txt
venv2/bin/pip install -r requirements-venv1.txt
user_92031100
Related