How to keep .m2 directory between jobs on Azure Pipeline?

Diego Cândido da Silva 0
Viewed 35

I have a pipeline where I'm trying to reuse the maven cache downloaded in the first job (.m2) in other jobs, but it seems that the azure pipeline deletes the files before each job. Is there any way to keep a folder and its files between jobs ?

my azure-pipeline.yml file:

when executing the tree command in the second job in the $(MAVEN_CACHE_FOLDER) directory, azure returns the following message: /home/vsts/work/1/a/.m2/repository [error opening dir]

pool:
  vmImage: ubuntu-latest

variables:
  MAVEN_CACHE_FOLDER: $(Build.ArtifactStagingDirectory)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

stages:
  - stage: validate
    displayName: VALIDATE
    jobs:
      - job: java_validations
        displayName: Java Validations
        workspace:
          clean: outputs
        steps:
          - checkout: self
            clean: false
          - task: Cache@2
            displayName: Cache Maven local repo
            inputs:
              key: 'maven | $(Agent.OS) | **/pom.xml, !**/target/**'
              restoreKeys: |
                maven | $(Agent.OS)
                maven
              path: $(MAVEN_CACHE_FOLDER)
          - script: |
              mvn validate $(MAVEN_OPTS)
            displayName: Maven Validate
          - script: |
              mvn checkstyle:check $(MAVEN_OPTS)
            displayName: Maven Checkstyle
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
      - job: unit_tests
        displayName: Unit Tests
        dependsOn:
          - java_validations
        workspace:
          clean: outputs
        steps:
          - checkout: none
            clean: false
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
          - task: Maven@3
            inputs:
              mavenPomFile: 'pom.xml'
              options: 'test-compile failsafe:integration-test -Dcheckstyle.skip -Pun-tests $(MAVEN_OPTS)'
              publishJUnitResults: false
              mavenVersionOption: 'Default'
              mavenAuthenticateFeed: true
              effectivePomSkip: false
              sonarQubeRunAnalysis: true
              sqMavenPluginVersionChoice: 'latest'
            env:
              JAVA_HOME: $(JAVA_HOME_17_X64)
              PATH: $(JAVA_HOME_17_X64)/bin:$(PATH)

UPDATE After answer of Kevin Lu-MSFT

Now I'm having problems with the maven cache, even downloading the cache and being able to see that the m2/repository folder exists, when calling the maven task the files are downloaded again, thus invalidating my attempt.

pool:
  vmImage: ubuntu-latest

variables:
  MAVEN_CACHE_FOLDER: $(HOME)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

stages:
  - stage: validate
    displayName: VALIDATE
    jobs:
      - job: java_validations
        displayName: Java Validations
        workspace:
          clean: outputs
        steps:
          - checkout: self
            clean: false
          - task: Cache@2
            displayName: 'Cache Maven local repo'
            inputs:
              key: '"funcs" | maven | "$(Agent.OS)" | pom.xml'
              restoreKeys: |
              path: $(MAVEN_CACHE_FOLDER)
          - script: |
              mvn validate $(MAVEN_OPTS)
            displayName: Maven Validate
          - script: |
              mvn checkstyle:check $(MAVEN_OPTS)
            displayName: Maven Checkstyle
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
      - job: unit_tests
        displayName: Unit Tests
        dependsOn:
          - java_validations
        workspace:
          clean: outputs
        steps:
          - checkout: self
            clean: false
          - task: Cache@2
            displayName: 'Cache Maven local repo'
            inputs:
              key: '"funcs" | maven | "$(Agent.OS)" '
              restoreKeys: |
              path: $(MAVEN_CACHE_FOLDER)
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
          - task: Maven@3
            inputs:
              mavenPomFile: 'pom.xml'
              options: 'test-compile failsafe:integration-test -Dcheckstyle.skip -Pun-tests $(MAVEN_OPTS)'
              publishJUnitResults: false
              mavenVersionOption: 'Default'
              mavenAuthenticateFeed: true
              effectivePomSkip: false
              sonarQubeRunAnalysis: true
              sqMavenPluginVersionChoice: 'latest'
            env:
              JAVA_HOME: $(JAVA_HOME_17_X64)
              PATH: $(JAVA_HOME_17_X64)/bin:$(PATH)
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
pipeline azure-pipelines azure-devops
1 Answers

Refer to this doc: Microsoft-hosted agents

Each time you run a pipeline, you get a fresh virtual machine for each job in the pipeline. The virtual machine is discarded after one job (which means any change that a job makes to the virtual machine file system, such as checking out code, will be unavailable to the next job).

From your YAML sample, you are using Microsoft-Hosted ubuntu agent. The two jobs are running on different virtual machines. So the files will not keep between jobs.

Here are the workarounds:

  1. You can create a Self-hosted agent. Then two jobs will run on the same agent and the files can be used between jobs.

  2. You can publish the files to Build artifacts in Job 1, and then you can download the files to the agent in Job 2. Then the files in Job 1 can be used in Job 2.

For more detailed info, you can refer to this doc: Publish and download pipeline Artifacts

Kevin Lu-MSFT0
Related