How to know which packages I istalled myself in conda (and pip)?

user_3261450 0
Viewed 514

It seems to me to be a very basic question, but I did not find a straightforward answer to it. I have two (in fact more) PCs where I work in conda environments. At some point I want to perform a major update, e.g. of Python version. Since conda indicates many conflicts, I did not know how to deal with, I could create a new environment with the desired Python version. But how should I know what packages shall I install in that new environment? In the 'old' one, I had:

  1. packages I installed by myself,
  2. packages conda found to be necessary,
  3. packages I installed via pip

So, when I create a new environment, how should I remember which packages I had installed in the old one, thus I should also install them in the new one? conda env export or conda list are not super helpful, as they display all packages (including their current versions and perhaps packages which are no longer valid with the new Python version) and I'd like to have filtered out only those that I myself had installed, so I would know they should also be present in the new environment (and the dependencies would be installed automagically). Same question applies to pip.

conda pip python
2 Answers

Conda has a specific term for environment constraints provided by user: explicit specifications. This would correspond to your first group of packages (requested by the user to be installed). One can use the --from-history flag to specify that the exported environment should only include explicit specs:

conda env export --from-history

Underneath it all, it processes the $CONDA_PREFIX/conda-meta/history file (which has the history of all env-mutating Conda commands) to distill out what constraints have been explicitly requested. It should also be noted that certain actions, such as using the --update-deps flag, can trigger Conda to add every package in the env to the explict specifications. Not sure of a clean way out of this; probably have to process the history file yourself to check for conda install commands.

None of this, however, will not capture things installed with pip.

user_5709180

+++ The verbose pip show command should show a line with the Installer. For example:

$ python -m pip show --verbose setuptools
Name: setuptools
Version: 49.6.0
[...]
Installer: pip
[...]

+++ The verbose pip list command should list all distributions active in the current environment and their installer. For example:

$ python -m pip list --verbose
$ .venv/bin/python -m pip list --verbose
Package       Version Location                                  Installer
------------- ------- ----------------------------------------- ---------
pip           20.2.3  /path/to/venv/lib/python3.8/site-packages pip
setuptools    49.6.0  /path/to/venv/lib/python3.8/site-packages pip

+++ For every distribution installed there should be a INSTALLER file. For example:

$ cat /path/to/venv/lib/python3.8/site-packages/setuptools-49.6.0.dist-info/INSTALLER
pip

+++ A short example code showing how to list all distributions in the current environment alongside their installer:

#!/usr/bin/env python3

import importlib.metadata

def main():
    for distribution in importlib.metadata.distributions():
        installer = None
        for file_path in distribution.files:
            if file_path.name == 'INSTALLER':
                installer = file_path.read_text().strip()
                break
        print(f"{distribution.metadata['Name']} -- {installer}")

if __name__ == '__main__':
    main()
user_111382590
Related