NUnit Test Run Order

Viewed 95770

By default nunit tests run alphabetically. Does anyone know of any way to set the execution order? Does an attribute exist for this?

16 Answers

Your unit tests should each be able to run independently and stand alone. If they satisfy this criterion then the order does not matter.

There are occasions however where you will want to run certain tests first. A typical example is in a Continuous Integration situation where some tests are longer running than others. We use the category attribute so that we can run the tests which use mocking ahead of the tests which use the database.

i.e. put this at the start of your quick tests

[Category("QuickTests")]

Where you have tests which are dependant on certain environmental conditions, consider the TestFixtureSetUp and TestFixtureTearDown attributes, which allow you to mark methods to be executed before and after your tests.

Wanting the tests to run in a specific order does not mean that the tests are dependent on each other - I'm working on a TDD project at the moment, and being a good TDDer I've mocked/stubbed everything, but it would make it more readable if I could specify the order which the tests results are displayed - thematically instead of alphabetically. So far the only thing I can think of is to prepend a_ b_ c_ to classes to classes, namespaces and methods. (Not nice) I think a [TestOrderAttribute] attribute would be nice - not stricly followed by the framework, but a hint so we can achieve this

Usually Unit Test should be independent, but if you must, then you can name your methods in alphabetical order ex:

[Test]
public void Add_Users(){}

[Test]
public void Add_UsersB(){}

[Test]
public void Process_Users(){}

or you can do..

        private void Add_Users(){}

        private void Add_UsersB(){}

        [Test]
        public void Process_Users()
        {
           Add_Users();
           Add_UsersB();
           // more code
        }

You should not depend on the order in which the test framework picks tests for execution. Tests should be isolated and independent. In that they should not depend on some other test setting the stage for them or cleaning up after them. They should also produce the same result irrespective of the order of the execution of tests (for a given snapshot of the SUT)

I did a bit of googling. As usual, some people have resorted to sneaky tricks (instead of solving the underlying testability/design issue

  • naming the tests in an alphabetically ordered manner such that tests appear in the order they 'need' to be executed. However NUnit may choose to change this behavior with a later release and then your tests would be hosed. Better check in the current NUnit binaries to Source Control.
  • VS (IMHO encouraging the wrong behavior with their 'agile tools') has something called as "Ordered tests" in their MS testing framework. I didn't waste any time reading up but it seems to be targeted towards the same audience

See Also: characteristics of a good test

In case of using TestCaseSource the key is to override string ToString method, How that works:

Assume you have TestCase class

public class TestCase
{
    public string Name { get; set; }
    public int Input { get; set; }
    public int Expected { get; set; }
}

And a list of TestCases:

private static IEnumerable<TestCase> TestSource()
{
    return new List<TestCase>
    {
        new TestCase()
        {
           Name = "Test 1",
           Input = 2,
           Expected = 4
        },
        new TestCase()
        {
            Name = "Test 2",
            Input = 4,
            Expected = 16
        },
        new TestCase()
        {
            Name = "Test 3",
            Input = 10,
            Expected = 100
        }
    };
}

Now lets use it with a Test method and see what happen:

[TestCaseSource(nameof(TestSource))]
public void MethodXTest(TestCase testCase)
{
    var x = Power(testCase.Input);
    x.ShouldBe(testCase.Expected);
}

This will not test in order and the output will be like this:

enter image description here

So if we added override string ToString to our class like:

public class TestCase
{
    public string Name { get; set; }
    public int Input { get; set; }
    public int Expected { get; set; }

    public override string ToString()
    {
        return Name;
    }
}

The result will change and we get the order and name of test like:

enter image description here

Note:

  1. This is just example to illustrate how to get name and order in test, the order is taken numerically/alphabetically so if you have more than ten test I suggest making Test 01, Test 02.... Test 10, Test 11 etc because if you make Test 1 and at some point Test 10 than the order will be Test 1, Test 10, Test 2 .. etc.
  2. The input and Expected can be any type, string, object or custom class.
  3. Beside order, the good thing here is that you see the test name which is more important.
Related