Previous    Next

The Pros and Cons

The use of dummy and mock objects is not uncontested, both in the software tester community and in the XP world. We tried to collect the most important arguments from both camps. We begin with the benefits of dummy objects:

• We can test with a finer granularity and higher accuracy. This shows that we can always trace a test failure back to an error in the test object or the test itself. This means that we won't have to dig deep into program layers that are currently of no interest.

• A dummy allows us to concentrate on the object under test and to create the initial state required for the test more easily. In contrast, building a complex initial state with the right (perhaps persistent) objects can represent a major problem.
• Dummy objects belonging to the test ensure reusability of our test. Real server objects may change their states due to a test and have to be reset after the test. This means additional effort in the best case; it might even be totally impossible.
• The use of real server objects in the test represents a kind of micro integration test. However, integration tests overlap heavily with function tests, and they are part of developer tests only in rare cases. Experience has also shown that tests integrating objects from many different layers can become too slow for a reasonably fast and continuous feedback.
• Dummy objects allow us to test the behavior of the test object along the margins of admissible value ranges. They also allow us to simulate error cases and exceptions in a very dedicated manner. Creating certain marginal conditions and error cases across several abstraction and access layers is very difficult and in some cases even impossible.
• In addition, dummy objects allow us to use a top-down approach in software development. Due to dependencies among objects, we would otherwise have to develop from layer to layer, in a bottom-up approach. This also means that, when using mock objects, we no longer have to define the complete infrastructure of our system upfront. Instead, we can build and expand it iteratively and incrementally.
• Dummy objects also help us test when a service provider we depend on is not yet implemented but has published its API.
• Testing with dummy objects improves the structure of the resulting program, because it prefers small objects and makes sure that the Dependency Inversion Principle and the Law of Demeter (see Glossary) are observed.

Mock objects are special dummy species, offering additional benefits:

• Conventional tests rely on the validation of return values and state changes of the test objects visible from the outside. Mock objects allow us to validate whether or not the test object's access to its helper and server objects is correct. This means that we actually test from the inside.

• Mock objects serve as containers, collecting duplicated test functionality. They facilitate refactoring of test code and represent a pattern that improves the communication ability of our code.

All points listed above relate to either increase of independence (of tests and code) or improvement of communication. All these benefits are confronted with drawbacks:

• Dummy classes can contain errors. However, this problem is relative because the probability that errors in the dummy class and errors in the test class will cancel each other out is small. Errors in the dummy class are normally discovered right away.

• Mock-based testing does not find errors resulting from the interplay of several components. We simply cover this error category by function tests. If we still find that such errors cause a frequent problem, then we should think about additional local integration tests in the critical places in the system. Ideally, they should integrate only two adjacent layers.
• Changes to the interface of the real implementation require changes to the dummy object. However, experience has shown that this additional effort accounts only for a small part of the total effort involved in updating all tests. The IDE also often helps find the signatures to be modified or extended.
• Using dummies for testing is something developers have to learn. However, both their experience and the reusable dummy and mock objects library increase over time.
• "Testing from the inside" means that we need to know what happens or should happen in the class. If the implementation changes, for example, because we found a better way to work with a server object, we also have to change the test code (including mock objects) frequently, even though the test object's behavior to the outside remained unchanged. For this reason, mock objects are used to test relatively stable implementations.

Robert Binder estimates the effort involved in creating stubs as being very high [Binder99, p. 662]. In particular, the large number of stub objects needed to supply each single test with the required responses is a big obstacle in the general use of this technique, according to Binder. Our experience does not confirm this theory; we normally manage with one single and easily configurable mock object per test. This discrepancy in experiences results partly from differences between development by the test-first or test-after approach, where the latter is based on the classic testing theory. Moreover, conventional stubs often simulate the real behavior of a system, requiring a much higher implementation effort, compared to slim mock objects.

Brian Marick [00, p. 110] identified two major problems in the use of stubs: (1) We implement each misconception we have about the real object in the dummy object. (2) Errors we would otherwise have found through indirect invocation in the helper object slip through. Reason (2) warns us against the assumption that, when using the mock technique, interaction tests like the ones described in , , can be totally omitted; at best, their number reduces.

Heuristics for the Use of Mocks

Careful balancing between the large number of benefits and drawbacks takes a lot of experience and the courage to experiment. We use dummy and mock objects in the following situations:

• We cannot do without them when the tests run too slowly, when the "right" class does not exist yet, or when certain boundary and error cases cannot be tested any other way.

• They improve the readability and maintainability of our test code, by removing duplicate code, for instance.

Whenever we can write the test as easily and clearly with the same or less redundancy without dummy objects, then we will do without them. The more we get used to them, the more often we will find good reasons to use them. One reviewer pointed out that it's common to start using a stub class for unimplemented functionality that evolves into the real class later on. This kind of temporary dummy is found most often during the development of model code. However, we have to be careful to ensure that our mock objects do not get too complex. Signs of excessive complexity include:

• They duplicate program logic from the "right" classes.

• They call other mock or dummy objects directly.
• We want to write test cases for mock objects ourselves.

In these cases, it makes sense to take a step back and ask ourselves whether we can simplify our mocks (e.g., by distributing them over several mock classes), whether we need them at all, whether a simple dummy class would do the trick, or whether the mock problem actually turns our attention to a design problem.

Previous    Next