Overview

Let's assume for the time being that the arguments in favor of unit tests and the test-first approach outlined in the introduction are true. One important prerequisite for this approach is the complete automation of the test process. The counter argument most frequently heard concerns the presumably higher initial cost, compared to manual tests. Is there something to this argument? In conventional test approaches it is assumed that the break-even point between costs and benefits is reached after about 10 test runs (). This means that automation pays at the latest when you run the automated test more than 10 times in unchanged condition. A rough estimate of our test-first approach looks like this:

With an average program size of 500 classes and a development time of one week per class, we have a total development time of 20,000 (500 x 5 x 8) hours. With an average integration frequency of hours and the rule that all unit tests have to be executed in each integration (see ), we have 5000 runs for the entire test suite. This means that a test from the middle of the development time is executed approximately 2500 times-provided it survives development to the end-not including that it may be executed every five to ten minutes during the initial focus of development. This automation definitely pays.

Naturally, this estimate does not include several factors: many tests have to be changed in the course of their lifetime, especially under constant refactoring (see , ); some tests will even disappear totally; and the cost for initial creation of a specific test can vary a lot.[] It also does not include additional effects of unit tests with regard to design and documentation. Still, our estimate is good enough to show that unit tests following the test-first approach would be much more expensive without automation (as measured in developer time), provided that the rules described in , , are meaningfully applied. In addition, humans produce many more errors than machines when doing recurring tasks over and over again, which is the case with the execution of most existing tests.

In summary, we cannot do without automation. The questions are now: What do we want to automate? How do we automate? How do we make sure we won't lose track? []A critical analysis of the economics of automated tests can be found in Marick [00].