Conda - how to update only cudatoolkit in an existing environment?

user_1757226 0
Viewed 3461

This is a specific instance of a general problem that I run into when updating packages using conda. I have an environment that is working great on machine A. I want to transfer it to machine B. But, machine A has GTX1080 gpus, and due to configuration I cannot control, requires cudatoolkit 10.2. Machine B has A100 gpus, and due to configuration I cannot control, requires cudatoolkit 11.1

I can easily export Machine A's environment to yml, and create a new environment on Machine B using that yml. However, I cannot seem to update cudatoolkit to 11.1 on that environment on Machine B. I try

conda install cudatoolkit=11.1 -c conda-forge

and I am met with ~5 minutes of conflicts and retrying solve messages, that ultimately don't really tell me anything useful (sorry, I did not capture that to post here, it is quite voluminous).

Short of re-creating the environment from scratch on Machine B, is there any way to update just cudatoolkit to the version that is required for that machine's GPUs?

I have also tried various permutations of conda update ... with no success.

If it helps, here is the yml file from Machine A:

name: VAE180
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.12.0=pyhd8ed1ab_0
  - aiohttp=3.7.4=py38h27cfd23_1
  - argh=0.26.2=pyh9f0ad1d_1002
  - async-timeout=3.0.1=py_1000
  - attrs=20.3.0=pyhd3deb0d_0
  - blas=1.0=mkl
  - blinker=1.4=py_1
  - brotlipy=0.7.0=py38h8df0ef7_1001
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h36c2ea0_0
  - ca-certificates=2020.12.5=ha878542_0
  - cachetools=4.2.1=pyhd8ed1ab_0
  - certifi=2020.12.5=py38h578d9bd_1
  - cffi=1.14.5=py38h261ae71_0
  - chardet=3.0.4=py38h924ce5b_1008
  - click=7.1.2=pyh9f0ad1d_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.2=pyhd8ed1ab_0
  - cryptography=3.4.6=py38ha5dfef3_0
  - cudatoolkit=10.2.89=hfd86e86_1
  - cycler=0.10.0=py38_0
  - dbus=1.13.18=hb2f20db_0
  - docker-pycreds=0.4.0=py_0
  - expat=2.2.10=he6710b0_2
  - ffmpeg=4.3=hf484d3e_0
  - fontconfig=2.13.1=h6c09931_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.7=pyhd8ed1ab_0
  - future=0.18.2=py38h578d9bd_3
  - gitdb=4.0.5=pyhd8ed1ab_1
  - gitpython=3.1.14=pyhd8ed1ab_0
  - glib=2.67.4=h36276a3_1
  - gmp=6.2.1=h2531618_2
  - gnutls=3.6.5=h71b1129_1002
  - google-auth=1.24.0=pyhd3deb0d_0
  - google-auth-oauthlib=0.4.1=py_2
  - gql=0.1.0=py_0
  - graphql-core=3.1.3=pyhd8ed1ab_0
  - grpcio=1.33.2=py38heead2fc_2
  - gst-plugins-base=1.14.0=h8213a91_2
  - gstreamer=1.14.0=h28cd5cc_2
  - gym=0.18.0=py38h81c977d_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyh9f0ad1d_0
  - importlib-metadata=3.7.3=py38h578d9bd_0
  - intel-openmp=2020.2=254
  - joblib=1.0.1=pyhd3eb1b0_0
  - jpeg=9b=h024ee3a_2
  - kiwisolver=1.3.1=py38h2531618_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libiconv=1.15=h63c8f33_5
  - libpng=1.6.37=hbc83047_0
  - libprotobuf=3.14.0=h8c45485_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.2.0=h3942068_0
  - libuuid=1.0.3=h1bed415_2
  - libuv=1.40.0=h7b6447c_0
  - libwebp-base=1.2.0=h27cfd23_0
  - libxcb=1.14=h7b6447c_0
  - libxml2=2.9.10=hb55368b_3
  - lz4-c=1.9.3=h2531618_0
  - markdown=3.3.4=pyhd8ed1ab_0
  - matplotlib=3.3.4=py38h06a4308_0
  - matplotlib-base=3.3.4=py38h62a2d02_0
  - mkl=2020.2=256
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.3.0=py38h54f3939_0
  - mkl_random=1.1.1=py38h0573a6f_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.4.1=hbb512f6_0
  - ninja=1.10.2=py38hff7bd54_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - nvidia-ml=7.352.0=py_0
  - oauthlib=3.0.1=py_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1j=h27cfd23_0
  - packaging=20.9=pyh44b312d_0
  - pandas=1.2.3=py38ha9443f7_0
  - pathtools=0.1.2=py_1
  - pcre=8.44=he6710b0_0
  - pillow=8.1.2=py38he98fc37_0
  - pip=21.0.1=py38h06a4308_0
  - promise=2.3=py38h578d9bd_3
  - protobuf=3.14.0=py38h2531618_1
  - psutil=5.7.3=py38h8df0ef7_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pyglet=1.5.15=py38h578d9bd_0
  - pyjwt=2.0.1=pyhd8ed1ab_0
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyparsing=2.4.7=pyh9f0ad1d_0
  - pyqt=5.9.2=py38h05f1152_4
  - pysocks=1.7.1=py38h578d9bd_3
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python_abi=3.8=1_cp38
  - pytorch=1.8.0=py3.8_cuda10.2_cudnn7.6.5_0
  - pytorch-lightning=1.2.4=pyhd8ed1ab_0
  - pytz=2021.1=pyhd3eb1b0_0
  - pyyaml=5.3.1=py38h8df0ef7_1
  - qt=5.9.7=h5867ecd_1
  - readline=8.1=h27cfd23_0
  - requests=2.25.1=pyhd3deb0d_0
  - requests-oauthlib=1.3.0=pyh9f0ad1d_0
  - rsa=4.7.2=pyh44b312d_0
  - scikit-learn=0.24.1=py38ha9443f7_0
  - scipy=1.6.1=py38h91f5cce_0
  - sentry-sdk=0.20.3=pyh44b312d_0
  - setuptools=52.0.0=py38h06a4308_0
  - shortuuid=1.0.1=py38h578d9bd_4
  - sip=4.19.13=py38he6710b0_0
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyh44b312d_0
  - sqlite=3.35.2=hdfb4753_0
  - subprocess32=3.5.4=py_1
  - tensorboard=2.4.1=pyhd8ed1ab_0
  - tensorboard-plugin-wit=1.8.0=pyh44b312d_0
  - tensorboardx=2.1=py_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - torchaudio=0.8.0=py38
  - torchvision=0.9.0=py38_cu102
  - tornado=6.1=py38h27cfd23_0
  - tqdm=4.59.0=pyhd8ed1ab_0
  - typing-extensions=3.7.4.3=0
  - typing_extensions=3.7.4.3=py_0
  - urllib3=1.26.4=pyhd8ed1ab_0
  - wandb=0.10.20=pyhd8ed1ab_0
  - watchdog=0.10.4=py38h578d9bd_0
  - werkzeug=1.0.1=pyh9f0ad1d_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - yaml=0.2.5=h516909a_0
  - yarl=1.6.3=py38h25fe258_0
  - zipp=3.4.1=pyhd8ed1ab_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - pytorch-lightning-bolts==0.3.0
prefix: /home/eric/miniconda3/envs/VAE180
conda package python
2 Answers

Overly-Restrictive Constraints

I'd venture the issue is that recreating from a YAML that includes versions and builds will establish those versions and builds as explicit specifications for that environment moving forward. That is, Conda will regard explicit specifications as hard requirements that it cannot mutate and so if even a single one of the dependencies of cudatoolkit also needs to be updated in order to use version 11, Conda will not know how to satisfy it without violating those previously specified constraints.

Specifically, this is what I see when searching (assuming linux-64 platform):

$ conda search --info 'cudatoolkit[subdir="linux-64",version="11.1.*"]'
Loading channels: done
...

cudatoolkit 11.1.1 h6406543_8
-----------------------------
file name   : cudatoolkit-11.1.1-h6406543_8.tar.bz2
name        : cudatoolkit
version     : 11.1.1
build       : h6406543_8
build number: 8
size        : 1.20 GB
license     : NVIDIA End User License Agreement
subdir      : linux-64
url         : https://conda.anaconda.org/conda-forge/linux-64/cudatoolkit-11.1.1-h6406543_8.tar.bz2
md5         : 4851e7f19b684e517dc8e6b5b375dda0
timestamp   : 2021-02-12 16:31:01 UTC
constraints : 
  - __cuda >=11.1
dependencies: 
  - __glibc >=2.17,<3.0.a0
  - libgcc-ng >=9.3.0
  - libstdcxx-ng >=9.3.0

Note how the dependencies conflict with your YAML specification. That is, the YAML locks libgcc-ng and libstdcxx-ng into versions 9.1.0, whereas cudatoolkit==11.1.1 requires versions 9.3.0 or greater.

Minimal Relaxing

It may be sufficient to edit only the specifications noted above. However, there is no simple way of telling if this might conflict with something else. It's worth a try though. That is, edit the YAML to have:

dependencies:
...
  - cudatoolkit=11.1
  - libgcc-ng
  - libstdcxx-ng
...

with the idea of leaving off the constraints on the dependencies, and let Conda solve them with whatever works.

Liberal Constraints

A more liberal way of exporting an environment is to use the --from-history flag:

conda env export --from-history -n VAE180 > VAE180.minimal.yaml

This will only export the explicit constraints from the user history, which may or may not include versions and builds, depending on how the user originally created the environment. Usually, it is far less constrained than what the default export will generate. One could then edit this YAML to have cudatoolkit=11.1 and then try to create the env with that.

The downside to this approach is that many of the other packages will likely take on newer versions, so it isn't as faithful a replication of the original environment as before.

user_5709180

Basically, I will always reinstall a new 11.1 Conda env for a new A100. Conda will install almost everything you need. Then, you can install extra packages for you projects. The count of extra packages is not that much for me. Even torch or tf env, or all scipy/pandas/sklearn stack would take you long.

user_75554320
Related