NUnit Throws exceptions but still passes

Viewed 22

I am running a new project with mstest with NUnit In Azure Devops pipeline. Locally No errors in output even when running it in command line: "vstest.console.exe" When I Push my commit to the git branch and run the pipeline the Test still pass but when I look at the logs I see some exception in the output.

System.IO.FileNotFoundException: Could not load file or assembly 'nunit.framework, Version=3.13.3.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies

I can choose to ignore them but that usually ends ups giving us trouble in the future. I am targeting net 4.7.2 this is because of a legacy project before we are able to upgrade. and I am using NUnit.3.13.3 & NUnit3TestAdapter.4.2.1.

My YAML:

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

steps:
- task: NuGetToolInstaller@1

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\NetLink.Test.dll
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

MY PIPELINE OUTPUT:

2022-09-06T18:32:55.2286691Z Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in D:\a\1\s\src\NetLink.Test\obj\Release\NetLink.Test.dll 2022-09-06T18:32:55.2287621Z An exception occurred in the driver while loading tests. 2022-09-06T18:32:55.2340151Z at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage) 2022-09-06T18:32:55.2341837Z at NUnit.Engine.Runners.DirectTestRunner.LoadPackage() 2022-09-06T18:32:55.2342578Z at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded() 2022-09-06T18:32:55.2343323Z at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter) 2022-09-06T18:32:55.2344047Z at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter) 2022-09-06T18:32:55.2345237Z at NUnit.VisualStudio.TestAdapter.NUnitEngine.NUnitEngineAdapter.Explore(TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\NUnitEngineAdapter.cs:line 88 2022-09-06T18:32:55.2346984Z at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 275 2022-09-06T18:32:55.2349381Z InnerException: System.IO.FileNotFoundException: Could not load file or assembly 'nunit.framework, Version=3.13.3.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. The system cannot find the file specified. 2022-09-06T18:32:55.2350958Z File name: 'nunit.framework, Version=3.13.3.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' 2022-09-06T18:32:55.2352646Z at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) 2022-09-06T18:32:55.2355042Z at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) 2022-09-06T18:32:55.2357386Z at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark) 2022-09-06T18:32:55.2358710Z at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo) 2022-09-06T18:32:55.2360814Z at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes) 2022-09-06T18:32:55.2362021Z at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes) 2022-09-06T18:32:55.2363179Z at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes) 2022-09-06T18:32:55.2364023Z at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args) 2022-09-06T18:32:55.2364626Z at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary2 settings) 2022-09-06T18:32:55.2365476Z at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage) 2022-09-06T18:32:55.2365997Z WRN: Assembly binding logging is turned OFF. 2022-09-06T18:32:55.2366517Z To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1. 2022-09-06T18:32:55.2367098Z Note: There is some performance penalty associated with assembly bind failure logging. 2022-09-06T18:32:55.2367650Z To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog]. 2022-09-06T18:32:55.2368120Z NUnit Adapter 4.2.0.0: Test execution complete

1 Answers

In this case, it seems the exception occurs when loading tests, so no tests are run.

In this case, the syntax of the VSTest runner may be helpful.

It has two options you can use:

  • failOnMinTestsNotRun: Fail the task if a minimum number of tests are not run. Default: False.
  • minimumExpectedTests: Optional. Use when failOnMinTestsNotRun = true. Minimum # of tests. Default: '1'.

So I would suggest updating your task as follows:

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\NetLink.Test.dll
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    failOnMinTestsNotRun: true

Which I think would likely have the desired effect.

Related