Process Types and Testing Strategies

With the process models used today, we can distinguish between different types, each requiring different testing strategies. Historically sequential models have been the first process models. They run tests at the end of the development project. In contrast, modern incremental and evolutionary development models require early integration and early testing of software pre-versions.

Sequential Models

Early process models were based on a sequential structure. The first and best known of these models is the waterfall model [, ], shown in .
Screenshot: The waterfall model .

The waterfall model is a sequential model because each of its activities takes place at a specific point within the process for the entire product. In a sequential model, all requirements are written and itemized within the requirement definition activity. At the end of the activity, the requirements are reviewed, coordinated, and specified. Similarly, the system and software design activity moves the design of the system architecture forward to the fine-grained design stage. Subsequently, the entire software is implemented, integrated, and tested. Finally, the software is used in its production environment, and later maintained and further developed. In sequential models, the integration of various modules and test runs takes place at the end of the corresponding phase, including the following partial activities:

Errors found during testing lead to regression within the process, as indicated by the dashed lines in . For example, if one of the tests finds an implementation error, then the implementation has to be corrected. Regressions within the process will be increasingly expensive, depending on how far back the cause of error is located within the process. For example, if a problem is due to a faulty requirement, then the requirement and the design based on it, as well as the program elements concerned, have to be modified and tested again. The linear structure of sequential models is plausible and facilitates planning and communication of a project. Each activity of a project leads to a defined intermediate result, normally in the form of a document (e.g., a requirement specification) during the early phases. Sequential models are found in many areas of our lives (e.g., in (building) architecture) where the technical process is well mastered. However, experiences gained in the past decades have shown that a sequential approach is seldom suitable for software development. The use of sequential models has led to problems in many software projects. This is mainly due to late integration and late testing of the software. Inconsistencies, misunderstandings, and quality problems often remain undiscovered till the end of a project, when the integration fails or testing exposes poor quality. Once problems have been found, work on the project has to start all over again, requirements have to be corrected, the design has to be modified, and large parts of the implementation have to be revised. This can mean that a project completed at "90%" is suddenly back where it all started .

Incremental Models

Incremental models are used to improve the technical manageability of projects and avoid the risk of a "big bang" integration at the end of a project [, , ]. shows the structure of such a process.
Screenshot: An incremental process.

In incremental processes, the system is built in a series of steps or increments. shows an example with three increments. To manage the project as a whole, incremental processes start with a coarse requirement definition for the entire project. A plan for the entire project describing all increments is based on this requirement definition. In addition, an architecture for the entire system is developed at the beginning of the project. It ensures that all increments can be realized and integrated into the entire system. Each increment creates an extension of the system, building on the previous step. One example for such an approach is the creation of a time entry system in three steps. Step 1 is a core system that reads data from input terminals, writes data consistently to a database, and produces basic reports about staff work hours. This core system is already mature enough to be tested and used. Step 2 adds functions to exchange data with the project planning and billing system; it also represents a testable and usable system. Step 3 is the entire system, essentially step 2 extended by a GUI for manual input and time corrections. Each single step is realized in an independent sequential process, often called a mini waterfall. The requirements are written in detail, a design is created, and new classes are implemented and tested in unit tests for each increment. Subsequently, the new classes are integrated into the entire system, followed by integration and system tests. The incremental development approach has the following implications on the testing strategy:

Similar to sequential processes, incremental processes require a basic understanding of the requirements for the entire project at the beginning of the project. Although the incremental approach is more flexible to change compared to sequential models, incremental models are not suitable for projects based on unclear or unknown requirements. An evolutionary approach is more appropriate for such projects.

Evolutionary Models

Evolutionary processes start with a vision of the product under development. However, the requirements are not known and cannot be gathered (by interviews or analysis of existing systems) at reasonable cost. [] In this situation, requirements can be stated only by realizing part of the product vision and the subsequent collection of feedback. This means that new preversions of the product are created iteratively, each covering an increasingly larger part of the vision. By using each pre-version in practice and implementing customer or endorser feedback, the product's pre-versions gradually approximate the customer's requirements. Although evolutionary development does not enable us to foresee the result of a project at the beginning, we have to plan each iteration. This plan describes the goals of each iteration (e.g., development of a central data repository component) and addresses project risks (e.g., technical feasibility in the chosen architecture). Each single iteration is implemented according to a suitable process model (e.g., a mini waterfall model), then the result is verified and feedback is collected. This evolutionary approach was described by Boehm [88] in a formal meta model- the spiral model. During the past years, many scientists and consultants have dealt with the question of how to practically use an evolutionary model with its frequent requirement and design changes. They have developed techniques to prevent the design and code from becoming unusable at some point in time due to constant changes. For this purpose, software developers need permission to change and improve the design of an existing (and runnable!) system, which conflicts with the principle, Don't fix it if it ain't broke. In his famous tutorial, Martin Fowler [99] used the term (see , ) to describe how such design and code renovations can be realized, provided we run a sufficient number of unit tests. The following rules apply to testing in the evolutionary process:

In the past few years there has been a rapidly growing interest in evolutionary models in the form of agile methodologies, which have stirred up interest all over the software landscape. Under the motto "agile," 17 representatives of modern, lightweight software methodologies joined efforts to formulate a common manifesto, the Manifesto for Agile Software Development . The processes (XP, Crystal, SCRUM, and Adaptive Software Development) proposed by the advocates of agile methodologies are evolutionary. The actual content of this manifesto concerns the process culture rather than the process itself. The basic notion is that the trained and motivated team, cooperating with the customer, comes first. Documents and tools are used only where they prove useful. The idea that a product can be fully specified in a document in advance is abandoned. Dramatic changes to the orientation of a project are seen as a characteristic of an increasingly dynamic business environment.

Continuous Integration

The trend away from sequential models towards incremental and evolutionary process models avoids late software integration. Following the thought through, we arrive at the idea to always keep the software in an integrated state, as XP requires. means integrating even small changes into the system, daily or even several times a day. []

Continuous integration translates into extreme requirements of the testing process. The system must not become inconsistent from integrating a changed class. This means that testing is necessary after each integration. It also means that automated regression tests are required, because no test team can constantly test their software. In addition, both the tests and the build procedure have to be very efficient. As far as the build is concerned, this results in the requirement for incremental compilation and a good library concept. As for the test, fast tests automated in the coding language with testing frameworks are the only feasible option (e.g., no GUI tests where time-consuming operation sequences are simulated). For example, CruiseControl is an open source tool that attempts to solve exactly these problems. []This is more often the case than you would expect. Unfortunately, these findings are often gained only at the end of a project. []Fowler and Foemmel discuss the advantages and realization of continuous integration extensively at .