How to manage development dependencies in Conda?

user_5104777 0
Viewed 546

I would like to have two environments: a production environment that is a strict subset of a testing environment. How can I achieve this using Conda (or micromamba)?

Testing requires some extra utilities that don't need to be bundled in the production docker image. The main idea is to ensure that the sub-dependencies in the production install are the exact same versions as were tested in the companion environment.

Example possibilities:

  • Is there an option to specify a constraints file (like for pip) that can include more packages than are currently being installed (and a way to populate this file from the test env, to use when creating the prod env)?
  • Is there some way to limit conda to only retrieving packages from a local cache that was freshly populated by the install of a previous env?
  • Is there an option to install the additional (testing) requirements while forcing conda not to upgrade/downgrade any existing packages in the environment? (But basing the test env from a duplicate of the prod env in this manner may make it impossible to satisfy version conflicts..)
continuous-deployment conda packaging
2 Answers

Going in the opposite direction (test -> prod) to my previous answer is a bit more tricky. Basically, we want to solve for the test environment first, and then only use package versions from that environment to create the production one. Below is one approach I checked will work, though it's not standard.

Temporary Package Cache + Offline Solve

The main idea is to use a temporary package cache so that only packages installed in the test environment would be available to the solve step when creating the pkg_prod environment. Here we go:

## create a temporary directory for package cache
tmp_pkg_cache=$(mktemp -d)

## create test environment (it uses both prod and test reqs)
CONDA_PKGS_DIRS=${tmp_pkg_cache} mamba create -n pkg_test \
  --file requirements.prod.txt --file requirements.test.txt

## restrict prod env creation to packages in the tmp cache
CONDA_PKGS_DIRS=${tmp_pkg_cache} mamba create -n pkg_prod \
  --file requirements.prod.txt --offline

## clean up cache
rm -rf ${tmp_pkg_cache}

Again, if requirements change, I'd remove the environments and rerun these steps. Note that this approach ignores existing package caches, so it will download every package - though if the runner is part of CI, that's probably the case anyway.

user_5709180

Conda doesn't directly support nesting environments in any useful way. However, there are well-defined steps that will yield environments with identical versions of common packages. Below is one possible procedure.

Clone + Frozen Install

To give a concrete example, and one that uses Conda in its generality (which is not Python-centric), let's consider a development project for an R package. Suppose we have the two sets of requirements:

requirements.prod.txt

r-base=4.0
r-magrittr
r-dplyr

requirements.test.txt

r-testthat
r-usethis

The following commands would create two environments, where the testing one (pkg_test) would be a superset of the production (pkg_prod) one:

## first, use Mamba if you aren't already
## conda install -n base conda-forge::mamba

## create the dev environment
mamba create -n pkg_prod --file requirements.prod.txt

## create the test environment as a clone
mamba create --clone pkg_prod -n pkg_test

## install the additional requirements, without updating pkgs
mamba install -n pkg_test --freeze-installed --file requirements.test.txt

If you change requirements, best would be to recreate the environments. I recommend Mamba because it minimizes solve time, provides more precise diagnoses when things go wrong, and rarely gives a surprising solve result.

user_5709180
Related