Azure DevOps Pipeline Global Package Installs Best Practices

user_14507103 0
Viewed 3654

For our pipeline we have two windows self-hosted agents installed on the same computer. Our main front-end pipeline .yml is listed below. This works fine except for some reason the npm install doesn't get nx, or jest. To fix this we can just run npm install -g nx and npm install -g jest once in the pipeline for each agent. After the first run it is fine and we can remove the extra installs to speed up execution. However, it won't update when new versions of nx or jest are released and this definitely isn't best practices.

I am guessing the problem is that these need to be install globally to work so the regular npm install can't achieve that. I included a slightly modified version of our package.json which shows that nx (@nrwl) and jest are included in the package. Does anyone know a better way to install the nx and jest without reinstalling them each pipeline build and without removing install statements after the first run with a new agent? Thank you in advance for any suggestions, please let me know if I can provide any additional information.


trigger:
- none

pool: 'myPool'
variables:
  npm_config_cache: $(Pipeline.Workspace)/.npm

steps:
- checkout: self
  clean: true

- task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  displayName: 'Install Node.js'

# Install stuff
- task: Npm@1
  timeoutInMinutes: 10
  inputs:
    command: 'install'
    workingDir: '$(System.DefaultWorkingDirectory)/ID_CLIENT'
  displayName: 'npm install'

# Test stuff
- script: nx affected:lint --base=origin/master --skip-nx-cache=true --parallel
  timeoutInMinutes: 10
  workingDirectory: '$(System.DefaultWorkingDirectory)/ID_CLIENT'
  displayName: 'run lint'

- script: 'jest --ci --reporters=default --reporters=jest-junit'
  timeoutInMinutes: 10
  workingDirectory: '$(System.DefaultWorkingDirectory)/ID_CLIENT'
  displayName: 'Run Tests'
  continueOnError: 'true'

# Build stuff
- script: nx affected:build --base=origin/master --skip-nx-cache=true --parallel
  timeoutInMinutes: 10
  workingDirectory: '$(System.DefaultWorkingDirectory)/ID_CLIENT'
  displayName: 'prod build'

# Publish test results to Azure Pipelines
- task: PublishTestResults@2
  timeoutInMinutes: 10
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/junit.xml' 
    searchFolder: '$(System.DefaultWorkingDirectory)'
    failTaskOnFailedTests: true

{
  "name": "project",
  "version": "1.0.0",
  "scripts": {
    "install:hard": "rimraf \"node_modules/!(rimraf)\" && npm cache clear --force && npm install",
    "install:globals": "npm install -g jest-cli @nrwl/cli",
    "postinstall": "ngcc"
  },
  "private": true,
  "dependencies": {
    "@angular/animations": "~9.1.4",
    "@angular/common": "~9.1.4",
    "@angular/compiler": "~9.1.4",
    "@angular/core": "~9.1.4",
    "@angular/forms": "~9.1.4",
    "@angular/localize": "^9.1.6",
    "@angular/platform-browser": "~9.1.4",
    "@angular/platform-browser-dynamic": "~9.1.4",
    "@angular/router": "~9.1.4",
    "@ng-bootstrap/ng-bootstrap": "^6.1.0",
    "@ngneat/until-destroy": "^7.3.0",
    "angular-datatables": "^9.0.1",
    "angulartics2": "^9.1.0",
    "bootstrap": "^4.4.1",
    "core-js": "^3.1.4",
    "jest-junit": "^12.0.0",
  },
  "devDependencies": {
    "@nrwl/angular": "^10.3.0",
    "@nrwl/jest": "^10.3.0",
    "@nrwl/workspace": "^10.3.0",
    "jest": "^26.4.2",
    "jest-canvas-mock": "^2.3.0",
    "jest-cli": "^26.4.2",
    "jest-preset-angular": "^8.3.1",
    "jest-skipped-reporter": "0.0.5",
    "ts-jest": "^26.4.1",
    "typescript": "^3.8.3",
  }
}

Edit: I tried adding "nx": "^10.3.0" and "@nrwl/cli": "^10.3.0" to the dev dependencies and then after installing I ran npm run nx test. It gave me npm ERR! missing script: nx. Did I just not add it correctly?

azure-pipelines npm node.js azure-devops
4 Answers

Well if you want to keep your global tools up to date and avoid running those command each time when your pipeline runs, create a new scheduled pipeline to run it once per day and to update your global tools.

Here you have docs about scheduled triggers.

But it may look like this:

schedules:
- cron: "0 0 * * *"
  displayName: Daily midnight build
  branches:
    include:
    - master
  always: true

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  displayName: 'Install Node.js'
- script: npm install -g jest-cli @nrwl/cli
user_23479990

The downside with automatically updating tools like that is things will break - without you knowing it - and rolling back is challenging.

Yes, it's best practice to have latest updates, esp. for bug fixes, etc. But medium term success also means stability via pinning (major/minor) versions is key when projects are in production. It's a bit more overhead to create your own images, but not much more if you're already using self-hosted agents.

NPX

Npx is intended to be used globally and since you are using self-hosted agents, create custom a custom agent image, pre-installed with the npx versions you want. Here is an example Azure Documentation about creating a custom image with Docker. You can also use a VM image.

Then in your pipeline, specify which one you want in your pipeline YAML:

pool: 'npx-9'

Others can choose a different version

pool: 'npx-10'

You can choose whatever image name you want. This lets developers choose what they need.

jest

Why do you need to install this globally?

Jest should be project specific, listed as a dev dependency in the project's package.json, and called via an npm script, e.g. npm run tests.

user_4665790

Nx has a script called .azure-pipelines/steps/install-node-modules.yml It caches the Node modules each run and checks to see if the cache was retrieved before it tries to npm install

This works fine but you will run into the same problems where outdated modules will be used ahead of the newer version.

Below is my modification of the npm install script:

- script: npm --prefix "$(System.DefaultWorkingDirectory)/Code/NxWorkspace" install "$(System.DefaultWorkingDirectory)/Code/NxWorkspace" --prefer-offline --no-audit --no-progress
  displayName: Install Node Modules

Instead of asking if the cache has been restored, I run:
npm install --prefer-offline --no-audit --no-progress

--prefer-offline prioritizes the cached packages, and will not try to install packages it can already find cached. If the version is bumped, the newer version is installed instead.

user_89486560

I ended up going with Max Ivanov's solution. We just added nx and the alias to the package.json and switched our pipeline scripts to use npm run nx ....

This has worked really well for us but I'd like to thank Krzysztof Madej and julie-ng for their answers. Both of which are very reasonable and would achieve what we wanted. I ended up not going with them because they don't fit our specific use case as well. If anyone is looking through this post I'd recommend looking at all the answers and picking the one that works best for you.

user_145071030
Related