Playwright test fails on Azure Devops Pipeline and pipeline hangs

ChunkyHippo 0
Viewed 42

I've created a simple pipeline in Azure Devops to run Playwright tests and explicitly coded one of the 2 tests to fail. When this happens during the pipeline execution, the pipeline seems to hang and I'm guessing will eventually timeout (after 30min, I cancelled). However when I execute tests that succeed, the pipeline exits successfully. Below is my yaml pipeline. Does anyone know why this might happen - have I missed something here?

[Edit]: I added the line below to my playwright.config.ts file and the task now fails like I expect. The default Playwright config opens the html report on any failures. There are a bunch of reporters you can use with Playwright, so take a look at what suits you (https://playwright.dev/docs/test-reporters).

reporter: [ ['html', { open: 'never' }] ],

trigger:
   - master
   - test
   - dev

pool:
   vmImage: macos-latest

jobs:
   - job: Build
     displayName: Build stage
     steps:
        - task: NodeTool@0
          inputs:
             versionSpec: '16.x'
          displayName: 'Install Node.js'
        - script: |
                npm ci
          displayName: 'Install dependencies'
        - script: |
            npx playwright install --with-deps
          displayName: 'Install Playwright'
        - script: |
            npm run test-unit-smoke
          displayName: 'Run unit tests'

enter image description here

azure-pipelines-yaml automated-tests playwright continuous-integration azure-devops
1 Answers

The reason is that your "CI" environment variable in the virtual server is not true at the moment you are running the tests. If CI is actually true at the moment, you will never see the "Serving HTML report.." text in your Pipeline logs.

I read online that CI providers should have CI as true by default, but at least in Azure Pipelines I've always had to manually set the value to true.

I haven't tried macOS, but with Windows I have to add "set CI=true" before running the tests in the YML-file.

Example with Azure Pipelines with a Windows VM:

- task: CmdLine@2
  displayName: 'Run all tests with Chrome and Firefox'
  inputs:
    script: 'set CI=true && npx playwright test ./tests/test.spec.ts --project=chromium --project=firefox'

With macOS you could try "export CI=true" before your test.

ttqa0
Related