JUnit5: How to repeat failed test?

user_142605 0
Viewed 12967

One of the practice many companies follow is to repeat unstable test until is passes x times (in a row or in total). If it is executed n times and fail to pass at least x times it is marked as failed.

TestNG supports that with the following annotation:

@Test(invocationCount = 5, successPercentage = 40)

How do I realize similar functionality with JUnit5?

There's similar annotation in JUnit5, called @RepeatedTest(5) but it is not executed conditionally.

junit5 java
4 Answers

U can try this extension for junit 5.

<dependency>
    <groupId>io.github.artsok</groupId>
    <artifactId>rerunner-jupiter</artifactId>
    <version>LATEST</version>
</dependency>

Examples:

     /** 
        * Repeated three times if test failed.
        * By default Exception.class will be handled in test
        */
       @RepeatedIfExceptionsTest(repeats = 3)
       void reRunTest() throws IOException {
           throw new IOException("Error in Test");
       }


       /**
        * Repeated two times if test failed. Set IOException.class that will be handled in test
        * @throws IOException - error occurred
        */
       @RepeatedIfExceptionsTest(repeats = 2, exceptions = IOException.class)
       void reRunTest2() throws IOException {
           throw new IOException("Exception in I/O operation");
       }


       /**
        * Repeated ten times if test failed. Set IOException.class that will be handled in test
        * Set formatter for test. Like behavior as at {@link org.junit.jupiter.api.RepeatedTest}
        * @throws IOException - error occurred
        */
       @RepeatedIfExceptionsTest(repeats = 10, exceptions = IOException.class, 
       name = "Rerun failed test. Attempt {currentRepetition} of {totalRepetitions}")
       void reRunTest3() throws IOException {
           throw new IOException("Exception in I/O operation");
       }

       /**
       * Repeated 100 times with minimum success four times, then disabled all remaining repeats.
       * See image below how it works. Default exception is Exception.class
       */
       @DisplayName("Test Case Name")
       @RepeatedIfExceptionsTest(repeats = 100, minSuccess = 4)
       void reRunTest4() {
            if(random.nextInt() % 2 == 0) {
                throw new RuntimeException("Error in Test");
            }
       }

View at IDEA:

IDEA looks like

With minimum success four times then disables all other: With minimum success four times then disables all other

You can also mix @RepeatedIfExceptionsTest with @DisplayName

source -> github

user_87165480

if you are running tests via Maven, with Surefire you care re-run failing tests automatically by using rerunFailingTestsCount.

However, as of 2.21.0, that does not work for JUnit 5 (only 4.x). But hopefully it will be supported in the next releases.

user_29879100

If you happen to be running your tests using the Gradle build tool, you can use the Test Retry Gradle plugin. This will rerun each failed test a certain number of times, with the option of failing the build if too many failures have occurred overall.

plugins {
    id 'org.gradle.test-retry' version '1.2.0'
}

test {
    retry {
        maxRetries = 3
        maxFailures = 20 // Optional attribute
    }
}
user_11083050
Related