Why don't PHPUnit flags order-by and stop-on work (the way I would expect them to)?

Viewed 492

I'm trying to get PHPUnit to stop on failure and to order tests by defect, in order to speed up my dev experience (no pun intended).

Here is what I try: I use "--cache-result --order-by=defects --stop-on-defect" on the command line, and additionally I use a phpunit.xml.dist with these flags:

<?xml version="1.0" encoding="UTF-8"?>

<!-- http://phpunit.de/manual/4.1/en/appendixes.configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.1/phpunit.xsd"
         backupGlobals="false"
         colors="true"
         bootstrap="app/autoload.php"

         cacheResult                    = "true"
         convertErrorsToExceptions      = "true"
         convertNoticesToExceptions     = "true"
         convertWarningsToExceptions    = "true"
         processIsolation               = "false"
         stopOnFailure                  = "true"
         executionOrder                 = "defects"
         beStrictAboutCoversAnnotation  = "true"
         beStrictAboutOutputDuringTests = "true"
         enforceTimeLimit               = "false"
>
[...]

But still my (second) test run looks like this:

[me@horus server]$ ./vendor/bin/phpunit --cache-result --order-by=defects --stop-on-defect
Runtime:       PHP 7.3.10
Configuration: /home/.../phpunit.xml.dist

PHPUnit 7.5.16 by Sebastian Bergmann and contributors.

S..S...........................................................  63 / 619 ( 10%)
...SSS......................................................... 126 / 619 ( 20%)
............................................................EEE 189 / 619 ( 30%)
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE.....................F

Time: 8.41 minutes, Memory: 483.04 MB

There were 43 errors:

[...]

I can tell it's really using my xml because when I mess up the format, the call to PHPUnit fails. But why are the tests not sorted to start with the first failure ? And why is there a whole line of failures, instead of just a single E ? Shouldn't it stop right after the first E ?

0 Answers
Related