A situation
I have a parameterized test method which works at all times. However, now I want to run this parameterized test in a parallel (every parameter set is a new independent test) and that's where my problems start.
What I have tried
At first I thought that maybe I could achieve my goal with a surefire plugin (3.0.0-M5)
and I've tested several configurations
1.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<forkCount>3</forkCount>
<parallel>all</parallel>
<reuseForks>false</reuseForks>
</configuration>
</plugin>
And this in my case worked like this
It takes 3 test classes at a time and execute them in parallel, however parameterized test is executed sequentially.
2.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<threadCount>3</threadCount>
<parallel>all</parallel>
</configuration>
</plugin>
Did not notice any changes related to a parameterized test (felt no difference from the first case).
Then I've tried Junit 5 parallel test functionality and surprisingly it started to parallelize a parameterized test, however random tests were failing due to incorrect output (input / predetermined output comparison failure) while others failed because of
org.mockito.exceptions.misusing.UnfinishedStubbingException:
Unfinished stubbing detected here:
-> at myStatefulService.process(...:85);
E.g. thenReturn() may be missing.
Examples of correct stubbing:
when(mock.isOk()).thenReturn(true);
when(mock.isOk()).thenThrow(exception);
doThrow(exception).when(mock).someVoidMethod();
Hints:
1. missing thenReturn()
2. you are trying to stub a final method, which is not supported
3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed
The configuration I've used while trying to parallelize a parameterized test with JUnit 5 was
junit-platform.properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=3
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.mode.classes.default=same_thread
I've also tried @DirtiesContext annotation with above mentioned methods, but to no avail.
My questions
Is it possible to parallelize a parameterized test using JUnit 5 or surefire plugin or a combination of both?
If yes, then how?
If no, then what options do I have?
My test structure that I'm trying to parallelize
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = {OtherClass.class})
public class myStatefulServiceTest {
@Autowired
private MyStatefulService myStatefulService;
@MockBean
private MyOtherService myOtherService;
public myStatefulServiceTest() {
}
@ParameterizedTest
@CsvFileSource(resources = "/my_test_input_file.csv", numLinesToSkip = 1)
public void processingTest(String pathToInputFile, String pathToValidatedOutput){
Message message = ...
doAnswer(s -> ...);
return null;
}).when(myOtherService).doAction(nullable(String.class));
try {
myStatefulService.process(message);
} catch (Exception ex) {
ArgumentCaptor<byte[]> response = ArgumentCaptor
.forClass(byte[].class);
verify(myOtherService, atLeast(1))
.doAction(nullable(String.class));
Assertions.assertEquals(..., ...);
}
}
}