Parallel testing when JUnit 5 + Surefire + @Parameterized test is used

Viewed 405

A situation

I have a parameterized test method which works at all times. However, now I want to run this parameterized test in a parallel (every parameter set is a new independent test) and that's where my problems start.

What I have tried

At first I thought that maybe I could achieve my goal with a surefire plugin (3.0.0-M5)

and I've tested several configurations

1.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <configuration>
        <forkCount>3</forkCount>
        <parallel>all</parallel>
        <reuseForks>false</reuseForks>
    </configuration>
</plugin>

And this in my case worked like this

It takes 3 test classes at a time and execute them in parallel, however parameterized test is executed sequentially.

2.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <configuration>
        <threadCount>3</threadCount>
        <parallel>all</parallel>
    </configuration>
</plugin>

Did not notice any changes related to a parameterized test (felt no difference from the first case).

Then I've tried Junit 5 parallel test functionality and surprisingly it started to parallelize a parameterized test, however random tests were failing due to incorrect output (input / predetermined output comparison failure) while others failed because of

org.mockito.exceptions.misusing.UnfinishedStubbingException: 
Unfinished stubbing detected here:
-> at  myStatefulService.process(...:85);

E.g. thenReturn() may be missing.
Examples of correct stubbing:
    when(mock.isOk()).thenReturn(true);
    when(mock.isOk()).thenThrow(exception);
    doThrow(exception).when(mock).someVoidMethod();
Hints:
 1. missing thenReturn()
 2. you are trying to stub a final method, which is not supported
 3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed

The configuration I've used while trying to parallelize a parameterized test with JUnit 5 was

junit-platform.properties

junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=3
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.mode.classes.default=same_thread

I've also tried @DirtiesContext annotation with above mentioned methods, but to no avail.

My questions

Is it possible to parallelize a parameterized test using JUnit 5 or surefire plugin or a combination of both?

If yes, then how?

If no, then what options do I have?

My test structure that I'm trying to parallelize

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = {OtherClass.class})
public class myStatefulServiceTest {

    @Autowired
    private MyStatefulService myStatefulService;

    
    @MockBean
    private MyOtherService myOtherService;
    
    
    public myStatefulServiceTest() {
    }

    @ParameterizedTest
    @CsvFileSource(resources = "/my_test_input_file.csv", numLinesToSkip = 1)
    public void processingTest(String pathToInputFile, String pathToValidatedOutput){
        
         Message message = ...
        doAnswer(s -> ...);
            return null;
        }).when(myOtherService).doAction(nullable(String.class));

        try {

            myStatefulService.process(message);

        } catch (Exception ex) {
            ArgumentCaptor<byte[]> response = ArgumentCaptor
                    .forClass(byte[].class);

            verify(myOtherService, atLeast(1))
                    .doAction(nullable(String.class));

            Assertions.assertEquals(..., ...);

        }
    }

}
1 Answers

You should use @MockitoSettings(strictness = Strictness.LENIENT) on the top of your test class

Related