Unit test is a part of daily coding, and once you have reached the critical mass, it is just not possible to go on without them. They spot bugs efficiently and systematically, and it is reassuring to have more and more of them. This is also why I started using a random suite for unit tests.
How to speed up unit tests
Naturally, the test suite will grow every other day, adding confidence and experience to it. But it will also reach a point where it start testing your patience and your resources : it just takes too long to run all the tests. Now, this is a sad milestone.
Until you set up a better environnement (dedicated servers, nightly/week end runs, paralell executions), an efficient way of getting results from a large unit test base is to run only a random suit of them. Out of a thousands unit tests, why not simply run a random selection of fifty ?
Extracting a random suite of unit tests
Using PHPUnit, I have a alltest suite, and I make this random selection based on it.
public static function suite() {
$suite = new PHPUnit_Framework_TestSuite('PHPUnit Framework');
$tests = ['test1.php', 'test2.php', 'test3.php', …. 'test1000.php'] ;
// the real list is MUCH longer, of course.
shuffle($tests);
$tests = array_slice($tests, 0, 50);
foreach($tests as $test) {
$suite->addTestSuite($test) ;
}
}
I am sure you may adapt this approach to your own unit test, may it be simple test or atoum.
Efficient unit testing
Now, this random list is short and will be run quite quickly. Knowing that all unit test must pass, any failure or error reported by the random selection has to be fixed.
Doing a short selection of test allows you to get some feed back faster. In the case of PHPunit, we may get a few failure, but will not have to wait for the whole list of test to be run. You may also use the option –stop-on-failure to have phpunit stop at the first failure. The above method may bring several failures in one quick batch.
Extra tips
- If you keep no track of the random tests run, the random suite is no more that the extra two lines above. Make sure you run the full suite sometimes (at night ? On week end ? ) to avoid missing some unlucky test.
- With some extra work, keep a list of unit test, result, number of execution and time of last execution in a SQL table. This will help you select the random tests by adding some priority to old tests, often failed one, least often tested one. Don’t forget to always use rand() in this process !
- Running random test is good for morale, and for any quick check during the day. It is not an alternative to running the full suite though. In fact, this suite is also undersized and should be bigger anyway !
the benefit of random
From recent experience, random tests is good at finding failed tests and it is definitely better than waiting for the next round of execution if it’s too slow.
It has the added benefit of yielding a shorter task list : less tests run, less failure found and thus, an affordable amount of work to fix them all.
It also seems to have the same effect than random passenger selection at the customs : running a few tests scares the other ones, which then yields less failures. Strange but true.