Azure DevOps code coverage with .coverage for PR and Cobertura for full report

user_909180 0
Viewed 1755

I have an Azure DevOps pipeline that validates pull requests. I have configured dotnet test to collect code coverage metrics using the --collect "Code coverage" argument:

- task: DotNetCoreCLI@2
  displayName: dotnet test
  inputs:
    command: 'test'
    arguments: '--configuration $(BuildConfiguration) --collect "Code coverage" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura'
    workingDirectory: $(baseWorkingDirectory)
    projects: 'tests/**/*.csproj'
    nobuild: true

As you can see, I'm also passing /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura. This I have to do in order to generete a coverage report:

- task: CmdLine@2
  inputs: 
    script: dotnet tool install -g dotnet-reportgenerator-globaltool

- task: CmdLine@2
  inputs: 
    script: reportgenerator -reports:$(Build.SourcesDirectory)/tests/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/CodeCoverage'

Documentation states that code coverage for pull requests is available only using the Visual Studio code coverage results format (file extension .coverage)

So:

  • I have to use Visual Studio code coverage --collect:"Code Coverage" to get code coverage for pull requests, because Cobertura format won't work.
  • I need to use Cobertura format in order to get a readable report on the Code Coverage tab in the pipeline, but the Cobertura report won't show up if I use --collect:"Code Coverage" at the same time.

It seems I can't get both code coverage for PRs and a full report in Cobertura format at the same time.

Other people seem to have the same problem, but the issue wasn't resolved in that thread.

Am I missing something?

azure-devops code-coverage
3 Answers

Can you try use this?

You have to install coverlet.collector package in all your test projects. Please notice that I used argument --collect:"XPlat Code Coverage".

# You just added coverlet.collector to use 'XPlat Code Coverage'
- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true'
    workingDirectory: $(Build.SourcesDirectory)

- task: DotNetCoreCLI@2
  inputs:
    command: custom
    custom: tool
    arguments: install --tool-path . dotnet-reportgenerator-globaltool
  displayName: Install ReportGenerator tool

- script: ./reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/coverlet/reports -reporttypes:"Cobertura"
  displayName: Create reports

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: $(Build.SourcesDirectory)/coverlet/reports/Cobertura.xml

Please be aware that you may have different folder strcture.

If you want to use Code coverage for pull requests feature you should not use Cobertura:

Code coverage for pull requests capability is currently only available for Visual Studio code coverage (.coverage) formats. This can be used if you publish code coverage using the Visual Studio Test task, the test verb of dotnet core task and the TRX option of the publish test results task. Support for other coverage tools and result formats will be added in future milestones.

user_23479990

In the pipeline definition:

- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    projects: 'path/to/test.csproj' # if multiple test projects are to be executed, create a dirs.proj and use it here. That will ensure the tests are executed in parallel
    arguments: '--collect "Code Coverage" '
    publishTestResults: true

In test project (or Packages.props if using https://github.com/microsoft/MSBuildSdks/tree/main/src/CentralPackageVersions)

...
  <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
...

According to https://github.com/microsoft/vstest/issues/981 and with Microsoft.NET.Test.Sdk 17.0.0+ coverlet is no longer needed as all coverage processing can be done using .coverage files on other-than Windows platforms using --collect:"Code Coverage". coverlet is no longer needed.

user_131981790

Unfortunately Azure DevOps is not supporting both code coverage PR support and readable reports at the same time.

My recommendation is to use dynamic code coverage (--collect "Code Coverage") and get *.coverage files. On PR build you can publish test results with publishRunAttachments: true. By this you should get code coverage PR support. Additionally you can use dotnet-coverage to convert your *.coverage reports to cobertura. Then you can use report-generator to generate readable coverage report. This coverage report can be uploaded as zip into artifacts of build. In this case using PublishCodeCoverageResults@1 will not work as Azure DevOps is not supporting it.

On non-PR builds you can again use dynamic code coverage but when publishing test results specify publishRunAttachments: false. Then again use dotnet-coverage to merge and convert your reports to cobertura. Finally you can add PublishCodeCoverageResults@1 task to get readable coverage report.

Both dotnet-coverage and report-generator are cross-plat dotnet global tools. You can easily install it in your pipeline.

Example script to generate html report and zip it:

dotnet-coverage merge -r -f cobertura -o merged.cobertura.xml *.coverage
reportgenerator -reports:merged.cobertura.xml -targetDir coverageReportHtml -reporttypes:HtmlInline
zip -r report.zip coverageReportHtml/*

There is one more workaround but I didn't try this. You could theoretically trigger 1 more build from your PR build. The second build could download *.coverage artifacts from first one and then use tooling like above to finally perform PublishCodeCoverageResults@1 task.

user_181099310
Related