How to make `dotnet test` output make sense with NUnit "Inconclusive"

user_19377819 0
Viewed 61

Struggling to understand how dotnet CLI is interpreting inconclusive tests and what to do about making it a bit more legible. When running a test project, inconclusive tests will be reported individually as skipped, but then the final summary will display zero skipped tests.

Skipped MyTest [61 ms]
Skipped MyTest [19 ms]
Skipped MyTest [22 ms]
Skipped MyTest [18 ms]
Skipped MyTest [20 ms]
Skipped MyTest [17 ms]
Skipped MyTest [15 ms]
Skipped MyTest [18 ms]
Skipped MyTest [14 ms]
Skipped MyTest [15 ms]
Skipped MyTest [16 ms]
Skipped MyTest [16 ms]

Passed!  - Failed:     0, Passed:    22, Skipped:     0, Total:    22, Duration: 1 s - MyTests.dll (net6.0)

Seeing 12 skipped tests in output and then 0 skipped in the summary, especially when I am expecting a total of 34 tests and only get 22 reported in total, is worrying. Why is it summarizing this way? Can this be configured somehow? Or can I get the individual test outputs to give a reason for skip to at least calm me down? (e.g. Skipped MyTest [16 ms] Reason: Inconclusive)

nunit .net c#
1 Answers

First the basics... Running dotnet test invokes the vstest runner, which assumes that any test will either pass or fail or show up as not run (skipped).

NUnit has an added outcome, Inconclusive, meaning that the test could not be completed because some required assumption was not met.

If you run your tests using dotnet test, those inconclusive tests have to be reported to the test runner through it's API, which only allows reporting of pass, fail or not run. Not run was chosen because, in fact, the test did not run to completion.

You can include a reason why the test is inconclusive just as you can give a reason for skipping or for a failure. The syntax would depend on how you are identifying a test as inconclusive in the first place... usually either Assert.Inconclusive or Assume.That.

If you make a lot of use of tests which can turn out inconclusive, it may be that dotnet cli isn't the best way to run them. For example, NUnit3-console would give you a tally including inconclusive tests and warnings in addition to those passed and failed.

user_15699670
Related