Dependency caching in Python CI pipeline in Azure DevOps?

user_4292396 0
Viewed 433

I followed the pip section on the Azure documentation on pipeline caching to speed up my Azure DevOps CI pipeline (in particular the dependency installation step). However, the packages are still installed every time I execute the pipeline (which I ideally also want to cache). How can I achieve this?

azure-pipelines conda pip azure continuous-integration
1 Answers

The Azure DevOps documentation is a bit lackluster here. Following the pip section just leads to caching of the wheels, not the installation itself (which you can also cache to further improve the pipeline execution time). To enable this, you need to work with a virtual environment (such as venv or a conda environment) and cache the entire environment.

Below you can find a code example with conda on how to cache an entire installed environment:

variables:
  CONDA_ENV_NAME: "unit_test"

  # set $(CONDA) environment variable to your conda path (pre-populated on 'ubuntu-latest' VMs)
  CONDA_ENV_DIR: $(CONDA)/envs/$(CONDA_ENV_NAME)

steps:
- script: echo "##vso[task.prependpath]$CONDA/bin"
  displayName: Add conda to PATH

- task: Cache@2
  displayName: Use cached Anaconda environment
  inputs:
    key: 'conda | "$(Agent.OS)" | requirements.txt'
    path: $(CONDA_ENV_DIR)
    cacheHitVar: CONDA_CACHE_RESTORED

- bash: conda create --yes --quiet --name $(CONDA_ENV_NAME)
  displayName: Create Anaconda environment
  condition: eq(variables.CONDA_CACHE_RESTORED, 'false')

- bash: |
    source activate $(CONDA_ENV_NAME)
    pip install -r requirements.txt
  displayName: Install dependencies
  condition: eq(variables.CONDA_CACHE_RESTORED, 'false')

# Optional step here: Install your package (do not cache this step)
- bash: |
    source activate $(CONDA_ENV_NAME)
    pip install --no-deps .
    pytest .
  displayName: Install package and execute unit tests
user_42923960
Related