removing Conda environment does not free up disk space

user_13769291 0
Viewed 1099

I installed a bunch of packages inside a Conda environment, including docker containers with NVIDIA toolkit and some other non-python libraries. The installation took about 10 GB of disk space. But when I removed the Conda environment, the disk space was not returned (not freed up). Shouldn't removing the environment lead to the removal of all files that were created inside that environment?

Some of the stuff that were installed inside the environment were installed by using pip, apt-get install, apt-add repository and so on.

When installing packages inside the Conda environment, the command line was prefixed by (my_environment).

The Conda environment was created on Ubuntu 18.04 using the following command: conda create -n my_environment

The environment was activated by: conda activate my_environment

The environment was removed by: conda remove --name my_environment --all

What is going on?

conda
1 Answers

Conda Package Caching

Conda works by first downloading packages to a package cache (check conda config --show pkgs_dirs) and then "installs" packages to specific environments by either copying from or hardlinking to the files/folders in the package cache. This procedure helps to minimize redundancy of packages across multiple envs. That is, it allows envs to share a single physical copy of the packages, when they use identical package versions (builds).

With that in mind, the answer is no, you are experiencing the expected behavior. Deleting an environment mostly just deletes hardlinks, but does not delete the files in the package cache. To accomplish the latter, one would need to additionally use

# read warning below before running this
conda clean --tarballs --packages

which will clear out any temporary tarballs and any packages that are not hardlinked to any environments.

Warning: Be aware of the flag description

 -p, --packages        Remove unused packages from writable package caches.
                       WARNING: This does not check for packages installed
                       using symlinks back to the package cache.

The warning doesn't pertain to most users (e.g., the default config is allow_softlinks: False), but some might be using symlinks (softlinks) to minimize redundancy in setups where the package cache is located on a different physical disk than where the environments are created.

user_5709180
Related