How to run test methods in specific order in JUnit4?

Viewed 455436

I want to execute test methods which are annotated by @Test in specific order.

For example:

public class MyTest {
    @Test public void test1(){}
    @Test public void test2(){}
}

I want to ensure to run test1() before test2() each time I run MyTest, but I couldn't find annotation like @Test(order=xx).

I think it's quite important feature for JUnit, if author of JUnit doesn't want the order feature, why?

23 Answers

If the order is important, you should make the order yourself.

@Test public void test1() { ... }
@Test public void test2() { test1(); ... }

In particular, you should list some or all possible order permutations to test, if necessary.

For example,

void test1(); 
void test2(); 
void test3(); 


@Test
public void testOrder1() { test1(); test3(); }

@Test(expected = Exception.class)
public void testOrder2() { test2(); test3(); test1(); }

@Test(expected = NullPointerException.class)
public void testOrder3() { test3(); test1(); test2(); }

Or, a full test of all permutations:

@Test
public void testAllOrders() {
    for (Object[] sample: permute(1, 2, 3)) {
        for (Object index: sample) {
            switch (((Integer) index).intValue()) {
                case 1: test1(); break; 
                case 2: test2(); break; 
                case 3: test3(); break; 
            }
        }
    }
}

Here, permute() is a simple function which iterates all possible permuations into a Collection of array.

JUnit since 5.5 allows @TestMethodOrder(OrderAnnotation.class) on class and @Order(1) on test-methods.

JUnit old versions allow test methods run ordering using class annotations:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
@FixMethodOrder(MethodSorters.JVM)
@FixMethodOrder(MethodSorters.DEFAULT)

By default test methods are run in alphabetical order. So, to set specific methods order you can name them like:

a_TestWorkUnit_WithCertainState_ShouldDoSomething b_TestWorkUnit_WithCertainState_ShouldDoSomething c_TestWorkUnit_WithCertainState_ShouldDoSomething

Or

_1_TestWorkUnit_WithCertainState_ShouldDoSomething _2_TestWorkUnit_WithCertainState_ShouldDoSomething _3_TestWorkUnit_WithCertainState_ShouldDoSomething

You can find examples here.

Its one of the main issue which I faced when I worked on Junit and I came up with following solution which works fine for me:

import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;

public class OrderedRunner extends BlockJUnit4ClassRunner {

    public OrderedRunner(Class<?> clazz) throws InitializationError {
        super(clazz);
    }

    @Override
    protected List<FrameworkMethod> computeTestMethods() {
        List<FrameworkMethod> list = super.computeTestMethods();
        List<FrameworkMethod> copy = new ArrayList<FrameworkMethod>(list);
        Collections.sort(copy, new Comparator<FrameworkMethod>() {

            @Override
            public int compare(FrameworkMethod f1, FrameworkMethod f2) {
                Order o1 = f1.getAnnotation(Order.class);
                Order o2 = f2.getAnnotation(Order.class);

                if (o1 == null || o2 == null) {
                    return -1;
                }

                return o1.order() - o2.order();
            }
        });
        return copy;
    }
}

also create a interface like below:

 @Retention(RetentionPolicy.RUNTIME)


@Target({ ElementType.METHOD})

public @interface Order {
public int order();
}

Now suppose you have class A where you have written several test cases like below:

(@runWith=OrderRunner.class)
Class A{
@Test
@Order(order = 1)

void method(){

//do something

}

}

So execution will start from method named "method()". Thanks!

JUnit 5 update (and my opinion)

I think it's quite important feature for JUnit, if author of JUnit doesn't want the order feature, why?

By default, unit testing libraries don't try to execute tests in the order that occurs in the source file.

JUnit 5 as JUnit 4 work in that way. Why? Because if the order matters it means that some tests are coupled between them and that is undesirable for unit tests.
So the @Nested feature introduced by JUnit 5 follows the same default approach.

But for integration tests, the order of the test method may matter since a test method may change the state of the application in a way expected by another test method.
For example when you write an integration test for a e-shop checkout processing, the first test method to be executed is registering a client, the second is adding items in the basket and the last one is doing the checkout. If the test runner doesn't respect that order, the test scenario is flawed and will fail.
So in JUnit 5 (from the 5.4 version) you have all the same the possibility to set the execution order by annotating the test class with @TestMethodOrder(OrderAnnotation.class) and by specifying the order with @Order(numericOrderValue) for the methods which the order matters.

For example :

@TestMethodOrder(OrderAnnotation.class) 
class FooTest {

    @Order(3)
    @Test
    void checkoutOrder() {
        System.out.println("checkoutOrder");
    }

    @Order(2)
    @Test
    void addItemsInBasket() {
        System.out.println("addItemsInBasket");
    }

    @Order(1)
    @Test
    void createUserAndLogin() {
        System.out.println("createUserAndLogin");
    }
}

Output :

createUserAndLogin

addItemsInBasket

checkoutOrder

By the way, specifying @TestMethodOrder(OrderAnnotation.class) looks like not needed (at least in the 5.4.0 version I tested).

Side note
About the question: is JUnit 5 the best choice to write integration tests? I don't think that it should be the first tool to consider (Cucumber and co may often bring more specific value and features for integration tests) but in some integration test cases, the JUnit framework is enough. So that is a good news that the feature exists.

JUnit 4 update

As of JUnit 4.13 @OrderWith, it is possible to reproduce the JUnit 5 @Order annotation. This solution better integrates with JUnit 4 than @RunWith a custom BlockJUnit4ClassRunner implementation.

Here's how I could replace method name ordering (@FixMethodOrder(MethodSorters.NAME_ASCENDING)) with an ordering by annotation.

@OrderWith(OrderAnnotation.class)
public class MyTest {
    @Test
    @Order(-1)
    public void runBeforeNotAnnotatedTests() {}

    @Test
    public void notAnnotatedTestHasPriority0() {}

    @Test
    @Order(1)
    public void thisTestHasPriority1() {}

    @Test
    @Order(2)
    public void thisTestHasPriority2() {}
}
/**
 * JUnit 4 equivalent of JUnit 5's {@code org.junit.jupiter.api.Order}
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD })
public @interface Order {
    /**
     * Default order value for elements not explicitly annotated with {@code @Order}.
     *
     * @see Order#value
     */
    int DEFAULT = 0;

    /**
     * The order value for the annotated element.
     * <p>Elements are ordered based on priority where a lower value has greater
     * priority than a higher value. For example, {@link Integer#MAX_VALUE} has
     * the lowest priority.
     *
     * @see #DEFAULT
     */
    int value();
}
import org.junit.runner.Description;
import org.junit.runner.manipulation.Ordering;
import org.junit.runner.manipulation.Sorter;

/**
 * Order test methods by their {@link Order} annotation. The lower value has the highest priority.
 * The tests that are not annotated get the default value {@link Order#DEFAULT}.
 */
public class OrderAnnotation extends Sorter implements Ordering.Factory {
    public OrderAnnotation() {
        super(COMPARATOR);
    }

    @Override
    public Ordering create(Context context) {
        return this;
    }

    private static final Comparator<Description> COMPARATOR = Comparator.comparingInt(
            description -> Optional.ofNullable(description.getAnnotation(Order.class))
                                   .map(Order::value)
                                   .orElse(Order.DEFAULT));
}

The not annotated tests get a default priority of 0. The order of tests with the same priority is undetermined.

Gist: https://gist.github.com/jcarsique/df98e0bad9e88e8258c4ab34dad3c863

Inspired by:

You can use one of these piece of codes: @FixMethodOrder(MethodSorters.JVM) OR @FixMethodOrder(MethodSorters.DEFAULT) OR @FixMethodOrder(MethodSorters.NAME_ASCENDING)

Before your test class like this:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class BookTest {...}

As others have stated, tests should be ideally be independent of execution order. This makes tests less fragile, and allows them to be run independently (many IDEs allow you to select a test method and execute it independently of other tests).

That being said, for integration tests, some people prefer to rely on method ordering.

Starting with JUnit 4.13 you can define your own class to reorder tests by extending Ordering. See the JUnit wiki for more details. Here's an example using the built-in Alphanumeric class to order the tests alphanumerically using the test method name:

import org.junit.Test;
import org.junit.runner.OrderWith;
import org.junit.runner.manipulation.Alphanumeric;

@OrderWith(Alphanumeric.class)
public class TestMethodOrder {

    @Test
    public void testA() {
        System.out.println("first");
    }
    @Test
    public void testB() {
        System.out.println("second");
    }
    @Test
    public void testC() {
        System.out.println("third");
    }
}

It's time to move to Junit5. Here is a sample of what we could get:

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
 class OrderedTests {

     @Test
     @Order(1)
     void nullValues() {}

     @Test
     @Order(2)
     void emptyValues() {}

     @Test
     @Order(3)
     void validValues() {}
 }

For Junit4, copy the logic that you have in several tests in one test method.

For JUnit 4, putting this annotation on the test class solved the problem.

@FixMethodOrder(MethodSorters.JVM)

I ended up here thinking that my tests weren't run in order, but the truth is that the mess was in my async jobs. When working with concurrency you need to perform concurrency checks between your tests as well. In my case, jobs and tests share a semaphore, so next tests hang until the running job releases the lock.

I know this is not fully related to this question, but maybe could help targeting the correct issue

If you want to run test methods in a specific order in JUnit 5, you can use the below code.

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class MyClassTest { 

    @Test
    @Order(1)
    public void test1() {}

    @Test
    @Order(2)
    public void test2() {}

}
Related