eXtreme Programming, or ‘XP’ as it is also known, is a relatively new development methodology (~4 years old), which treats software development as a ‘jigsaw’ puzzle: a product is slowly built by adding small pieces to a larger body of work. The methodology is designed to deliver the software that the customer needs when it is needed, accommodating changing requirements, even late in the life cycle.
The four sought-after values in eXtreme Programming are: Communication, Simplicity, Feedback, and Courage. Testing needs to support Development in these values to allow them to move forward with what they have to do, which is to implement the current iteration’s functionality. Development needs to know that collateral damage caused in the current iteration will be caught immediately, before it is time to release.
The goal of this article is not to talk about XP… directly – there are lots of other articles and sites out there about eXtreme Programming that you can read. This site is about how to test software that is developed following XP principles; using a methodology we can call eXtreme Testing.
The biggest challenge of testing, regardless of the methodology, has always been to cover everything with the resources and time available. The intent of eXtreme Testing is to directly address this issue but in the framework of eXtreme Programming. eXtreme Testing is designed to allow testers to get ramped up on the project as quickly as possible and then to allow them to keep up after that. The values of eXtreme Programming are directly applicable to Xtreme Testing.
The general approach of eXtreme Testing is to define a straight-forward test strategy for the iteration. Prioritize the acceptance tests (ATs) defined by the client and any complementary tests necessary. Automate what can be automated. And, ensure that each release has its new functionality ‘Smoke Tested’ and that no regression in previously existing functionality has occurred. The test activities must be efficient, well structured, easy to run, and the results easy to communicate. The focus of eXtreme Testing is to be confident that what needs to be tested has been without slowing the project.
You want to try eXtreme Testing on your project? Add a little to your current methodology or try it all at once. The basic principles described here can benefit any project, XP or not.
Workflow
The eXtreme Testing workflow is seen to ‘lag’ slightly behind the development cycle. Thus, if a project is estimated to have 10 iterations of development, there will need to be (a minimum of) 10 iterations of testing (test planning, automation, test), but they may lag by up to one iteration, making the complete project length 11 iterations.
NOTE: Depending on the type of project, full automation of testing may not make sense or even be possible.
The eXtreme Testing workflow is comprised of several parallel streams of activity (A to F in the diagram below), including Project Planning and Development:
- In each iteration, the development team and the client agree to the features or stories (A) that will be implemented and delivered by the end of the iteration. Out of this negotiation, the client will define the ATs for the coming release.
- Testing will perform a high-level test analysis of the stories and incorporate that analysis and the client’s ATs into a simple test plan (C) for the iteration.
NOTE: The test analysis, when presented to the client, may result in modification of the ATs.
NOTE: In the initial iteration, more thought and effort must be put into developing the test strategy than with successive iterations, much the same as when defining the project architecture. - As part of the test planning component of the iteration, estimates of effort required to automate the ATs and execute functional testing are created and compared against estimates from previous iterations. The test effort is then prioritized to accomplish as much AT automation and functional testing as possible within an iteration’s timeframe. The remaining tests will be addressed at the beginning of the next iteration.
- While development (B) of the current iteration’s functionality is being undertaken, automation of the ATs (D) of the current iteration’s developed functionality is started (in order of descending priority), and automation of any unfinished or defective ATs from the previous iteration is completed. The early automation of new ATs will allow defects to be uncovered and corrected on an on-going basis.
NOTE: The automation of the new ATs may not all be available until after the smoke test for the current iteration, but *some* of the automation could be run on the current iteration and the remainder of the tests executed manually. - While development of the current iteration’s functionality is being completed, functional testing (E) of the previous iteration’s developed functionality and regression of corrected defects is on-going. The problem reports resulting from this testing will be prioritized and addressed accordingly.
NOTE: Some problem reports could be postponed until the next iteration or perhaps even longer. - Integration of the current iteration’s functionality should be followed by a Smoke Test of the new functionality paired with regression testing of corrected defects. This allows a sanity check of the integrated components and how they are working together. The Smoke Test will be comprised mainly of the ATs for the current iteration and critical ATs from previous iterations.
NOTE: Complete functional and regression testing will occur at the beginning of the next iteration. - Release (F) of the current, tested, iteration is made to the customer. This small release is designed to generate feedback on the requirements and functionality as implemented to date and is used as an input to the Feature Set Definition component of the next iteration.
- Re-Iterate.
Integrating Testing with eXtreme Programming
What about other sorts of testing?
Performance, Stress, and Usability testing are just a few of the other sorts of tests that may be appropriate to the project. Time and resources need to be scheduled to undertake these activities at sensible points in the project life cycle. In some cases it may make sense to devote an iteration to these types of tests while Development takes the opportunity to catch up on outstanding problem reports.
How does eXtreme Testing meet the ‘rules’ of eXtreme Programming?
The basic set of testing rules for eXtreme Programming are covered by the following rules of eXtreme Testing:
- Unit Tests must be documented for each component
- All Acceptance Tests (ATs) must be documented
- Uncovered defects must have Regression Tests created to guard against its return
- All Unit Tests must pass before a component is allowed to be integrated
- All ATs and Regression Tests must pass before a Release is made
- Automate any test when it makes sense and where it is possible
More on eXtreme Programming?
Where can you get more information on eXtreme Programming? There are links to projects, conferences, books, web sites, and people that can be of help.
