Mutation Testing

Do you write code ?

Do you test your code ?

Do you test your tests ?

Do you test the tests for your tests ?

Testing tests ?

Richard Lipton, Fault Diagnosis of Computer Programs, 1971

Wrap up

The good parts

Detect problems with your tests
They discover dead code
They are automatic
How else would you test your tests ?
(Semi-)Automatic tool for testing ? I'm in !

What could be the advantages of automatic mutation testing tools ?
They can help you detect problem with your tests
But they can also help you detect problems with your code, like a dead code
They are automatic. You won't write the mutation tests by hand because it would take too much time.
You will either use an existing library or at least you will write your own library.
And let's face it - how else would you measure the quality of your tests ?
We have test coverage statistics for the code, but we don't have anything like that for tests.
So, if there was a good tool, that I could plug into my continuous integration cycle that would report problems with my test, then I'm totally sold.
Studies shows that software developers spend up to 50% of their time just on testing. I would love to have a tool that gives me automatic feedback on my tests.

The not so good parts

Mutation testing is slow:
(TIME = ALL MUTANTS x ALL TESTS)
Handful of libraries
Equivalent Mutant Problem
Writing complex mutant tests is difficult

And that bring us to the "not so good parts".
First of all - mutation testing is slow. If you think that your whole test suite is slow because it runs for 2 minutes, then imagine that proper mutation testing requires running all your tests for the all the mutants. Well, there are some studies on how to speed up this process, like using selective mutations or mutant sampling, but they all boil down to running less tests and that might overlook some bugs.
There are not so many good libraries for mutation testing.
This is often caused by the aforementioned Equivalent Mutant Problem, which is very difficult to solve.
Also, writing complex tests, beyond the simple operator changing or variables replacement, it's basically impossible.

Mutant testing libraries

Mutant - Ruby (last updated September 2015)
VisualMutator - C# (last updated September 2015)
Pitest - Java (last updated August 2015)
Humbug - PHP (last updated May 2015)
MuCheck - Haskell (last updated January 2015)
MutPy - Python3 (last updated January 2014)
Mutator - commercial solution for Java, Ruby, JavaScript and PHP

The future ?

Thank you !

Any questions ?

Happy ~~coding~~ testing !

This presentation is available on github, so you can see the slides on github pages