Is mutation testing useful in practice?

The usefulness of unit tests is no longer discussed. They are essential in conception of a quality application. But, how can we assess their relevance? A code coverage indicator up to 100% doesn’t mean the code is 100% tested. This is just a view of executed code during unit tests execution. Mutation testing will allow you to have more confidence in your tests.

This is a two step process:

1. Generate mutants.
2. Check that the mutations are found by the tests.

I wrote a entire article about this process, including some concrete cases.

I looked at mutation test some time ago as a method for checking the efficacy of my automated regression testing scripts. Basically, a number of these scripts had missing checkpoints, so while they were exercising the application being tested correctly, they weren't verifying the results against the baseline data. I found that a far simpler method than changing the code was to write another application to introduce modifications to a copy of the baseline, and re-run the tests against the modified baseline. In this scenario, any test that passed was either faulty or incomplete.

This is not genuine mutation testing, but a method that uses a similar paradigm to test the efficacy of test scripts. It is simple enough to implement, and IMO does a good job.

Mutation testing has helped me identify problems with test case assertions.

For example, when you get a report that says "no mutant has been killed by test case x", you take a look, and it turns out the assertion had been commented out.

According to this paper, developers at Google use Mutation testing as a complement to code-review and pull-request inspections. They seem happy about the results:

Developers have decided to redesign large chunks of code to make them testable just so a mutant could be killed, they have found bugs in complex logical expressions looking at mutants, they have decided to remove code with an equivalent mutant because they deemed it a premature optimization, they’ve claimed the mutant saved them hours of debugging and even production outages because no test cases were covering the logic under mutation properly. Mutation testing has been called one of the best improvements in the code review verification in years. While this feedback is hardly quantifiable, combined with the sheer number of thousands of developers willing to inspect surfaced mutants on their code changes makes a statement.

Coverage vs mutation testing. An old question, but I recently came across a recent blog on the topic. Pretty opinionated. But the differences between coverage and mutation testing is clearly articulated.

https://pedrorijo.com/blog/intro-mutation/

My own experience shows that Pitest is pretty useful, but since the runtime explodes it works only one very fast test sets. In practice this limits where I apply mutation testing.

The test case for the first one behaves differently due to above mutation there is an exception raised now. So it doesn’t returns the expected array of {6,3}. However, our second test case remains same, because it also includes positive number. So, it gives exception on positive numbers as well. Now, if we have to write a successful test case that would be Input ={-6,-6,-7,-3,-4} Expected = {-6,-3}

I set up mutation testing on Angular using https://stryker-mutator.io/docs/stryker-js/guides/angular/ simply to experiment and it took 2 hours to get a report for a single code file. That said, I was very happy with the experience of using Stryker with .NET. I must admit I am fairly new to mutation testing and there might be better tools that work with Angular/karma but performance is something to keep in mind especially if you plan to use it in conjunction with TDD.