In the functional tests of a Symfony5.2 application, I added some profiling assertions to check execution time.
private static function assertTime(?Profile $profile, int $milliseconds = self::MAX_TIMES)
{
if (null === $profile) {
static::markTestSkipped('To test time of this use case, please unable profiling');
}
/** @var TimeDataCollector $time */
$time = $profile->getCollector('time');
static::assertLessThan(
$milliseconds,
$time->getDuration(),
sprintf(
'Checks that execution time (%dms) is less than %dms (token %s)',
$time->getDuration(),
$milliseconds,
$profile->getToken()
)
);
}
It works fine.
But each time I launch my testsuite, the first test failed because of execution time:
symfony php ./bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
Runtime: PHP 7.4.6 with Xdebug 2.9.5
Configuration: /home/alexandre/PhpstormProjects/bb-one/phpunit.xml.dist
Testing
F......................... 26 / 26 (100%)
Time: 00:19.352, Memory: 82.50 MB
There were 1 failure:
1) App\Tests\Functional\MailTest::testShowArchived
Checks that execution time (2952ms) is less than 1000ms (token b6742a)
Failed asserting that 2952.1640625 is less than 1000.
I know this is the first functional test, because, if I change default order, I have:
symfony php ./bin/phpunit --order-by reverse
There were 1 failure:
1) App\Tests\Functional\MailTest::testShowTrash
Checks that execution time (2868ms) is less than 1000ms (token a3342a)
Failed asserting that 2868.4316425 is less than 1000.
- I tried to purge then warmup cache, before launching testsuite but it changes nothing.
- I read this answer, but it isn't a good answer.
- I tried to start the server in test mode, no change.
- I tried to create a kernel in the bootstrap to call the console and clear the test cache (it didn't help)
- I tried to create an initial test like
AAATest::testAAA(){/*..some functional test without profiling assertion ..*/}, this solution is acceptable when we launch our full tests as you can see in the below output:
Testing
........................... 27 / 27 (100%)
Time: 00:20.783, Memory: 76.50 MB
OK (27 tests, 174 assertions)
But when a developer is optimizing a specific use case of the site, he only launchs the corresponding test, and the time is increased by startup and the test failed.
I find a bad solution to avoid that the first one failed during the full testsuite. Now, I'm searching a solution to avoid that my test failed when I launch only one test!