Costs and Benefits of Automated Unit Tests

The main benefit of tests is to prevent and find errors. Errors found during the development phase mean cash money. The removal of an error found during the development is much cheaper than removing the same error when the software has already been delivered. Errors found after delivery lead to unhappy customers, standstill of equipment or business processes, difficult and expensive error localization, and complex problem tracking and remove processes, and eventually to new delivery and installation of the software. In addition to the enormous savings potential of early error identification, unit tests offer more benefits. Unit tests created by the test-first approach have a strong positive influence on the design. Moreover, unit tests are local, therefore the cause of errors is normally found in the tested class. If an error occurs in the system test, then localizing the error in the entire system is much more expensive. And additional costs (e.g., traveling expenses, standstill of business processes, penalties, etc.) have to be expected if the software is already in productive use. The benefit of unit tests can be explained by the numbers: Studies conducted by IBM [Subramaniam99] observed that an error found during the coding or unit testing phase is cheaper by a factor of 10, compared to when it is found in the system test. This factor rises to 100 for errors found as late as during the software's productive use. At the other end of this enormous benefit are the costs for creating and updating tests. These costs are higher for automated than for manual tests. In contrast, the costs for test runs are much higher in manual testing. Screenshot shows that automated tests pay off if they are repeated many times. From a specific number of n iterations, the costs for the n-th manual test run are higher than the costs of the n-th automated test run, plus one-time costs for creating the automated test.

Java Click To expand
Screenshot: Comparing the costs of manual versus automated testing.

Another cost factor in automation is the cost for test reworking. When the functionality or the public interface of a class changes, then we also have to adapt our test cases. This means that, in reality, the cost for test automation is not constant; it depends on the (hard to foresee) rate in which the software changes. Despite the difficulties in estimating the exact costs, we can state the following:

[3]The program is additionally put through a "smoke test," a relatively simple check, to see whether the product starts to smoke when being switched on.