Download conda data science libraries without extracting the packages

user_15196234 0
Viewed 800

I want to create a Python environment with the data science libraries NumPy, Pandas, Pytorch, and Hugging Face transformers. I use miniconda to create the environment and download and install the libraries. There is a flag in conda install, --download-only to download the required packages without installing them and install them afterwards from a local directory. Even when conda just downloads the packages without installing them, it also extracts them.

Is it possible to download the packages without extracting them and extract them afterwards before installation?

data-science conda download
2 Answers

There is no simple command in the CLI to prevent the extraction step. The extraction is regarded as part of the FETCH operation to populate the package cache before running the LINK operation to transfer the package to the specified environment.

The alternative would be to do something manually. Naively, one could search Anaconda Cloud and manually download, however, it would probably be better to go through the solver to ensure package compatibility. All the info for operations to be run can be viewed by including the --json flag. This could be filtered to just the tarball URLs and then downloaded directly. Here's a script along these lines (assuming Linux/Unix):

File: conda-download.sh

#!/bin/bash -l
conda create -dn null --json "$@" |\
    grep '"url"' | grep -oE 'https[^"]+' |\
    xargs wget -c

which can be used as

./conda-download.sh -c conda-forge -c pytorch numpy pandas pytorch transformers

that is, it accepts all arguments conda create would, and will download all the tarballs locally.

Ignoring Cached Packages

If you already have some packages cached then the above will not redownload them. Instead, if you wish to download all tarballs needed for an environment, then you could use this alternate version which overrides the package cache using an empty temporary directory:

File: conda-download-all.sh

#!/bin/bash -l
tmp_dir=$(mktemp -d)

CONDA_PKGS_DIRS=$tmp_dir conda create -dn null --json "$@" |\
    grep '"url"' | grep -oE 'https[^"]+' |\
    xargs wget -c

rm -r $tmp_dir
user_5709180

Do you really want to use conda-pack? That lets you archive a conda-environment for reproducing without using the internet or re-solving for dependencies. To just prevent re-solving you can also use conda env export --explict but that still ties you to the source (internet or local disk repository).

If you have a static environment (read-only) and want to really reduce docker size, you can volume mount the environment at runtime. You would need to match the file paths (ie: /opt/anaconda => /opt/anaconda).

user_2689070
Related