Run GitlabCI goals without recompiling at every stage

user_10374425 0
Viewed 688

I have a modular Java project hosted on Gitlab, with some Gitlab CI to check the code everytime we push some changes. The problem is that the project grew quite a lot recently, and every CI pipeline now takes about 15 minutes to complete.

What I noticed is that the code is basically compiled three times: during the "build" job (which runs mvn compile), during the "test" job (which runs mvn test) ans during the "deploy" job (which runs mvn install). Also, the tests were run twice, once in "test" and once in "deploy".

I tried to change this behavior by transforming the mvn test into mvn resources:testResources compiler:testCompile surefire:test and mvn install into mvn jar:jar install, but now the tests fail because of unknown classes and packages (even if it does work when I try to run all the commands one after another manually), like package my.project.package does not exist.

Maybe am I trying to do something that Gitlab CI can't do ? Maybe there is a better way to do it ?

Here is my YAML file:

image: myproject:latest

variables:
  MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true -U"

cache:
  key: new_cache
  paths:
    - .m2/repository/
    - target/

build:
  stage: build
  tags:
    - shell
  script:
    - mvn $MAVEN_CLI_OPTS clean compile

test:
  stage: test
  tags:
    - shell
  script:
    - mvn $MAVEN_CLI_OPTS resources:testResources compiler:testCompile surefire:test


install:
  stage: deploy
  tags:
    - shell
  script:
    - mvn $MAVEN_CLI_OPTS jar:jar install:install
  only:
    - master
gitlab-ci maven java
2 Answers

Apparently the default behavior of Gitlab CI is to delete the result of a job after it has been completed, but there are two features called artifacts and dependencies which allow a user to save some of the data and transmit it to other jobs. Now, the file looks like this:

image: myproject:latest

variables:
  MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true -U"

cache:
  key: new_cache
  paths:
    - .m2/repository/
    - frontend/node_modules/

build:
  stage: build
  tags:
    - shell
  script:
    - mvn $MAVEN_CLI_OPTS clean compile
  artifacts:
    paths: 
      - backend/target/

test:
  stage: test
  tags:
    - shell
  script:
    - mvn $MAVEN_CLI_OPTS resources:testResources compiler:testCompile surefire:test
  dependencies:
    - build

install:
  stage: deploy
  tags:
    - shell
  script:
    - mvn $MAVEN_CLI_OPTS jar:jar install:install
  dependencies: 
    - build
  only:
    - master

Now, the test and deploy jobs have access to the target directory and can run without recompiling everything. Combined with the caching of node_modules, it reduced the execution time to just 5 minutes.

user_103744250

The best approach is not to split the Maven build into parts, but just run

mvn clean install

or

mvn clean deploy

without running the compile and test phases separately before.

user_9274930
Related