Write a Test Strategy – What Choice(s) Do I Have?

Project Manager: “Your test strategy looks good.”    
    Test Lead: “Great. I will get started on the next steps. Do you have the finalized delivery schedule?”
Project Manager: “I will send it to you shortly.”    

What is a Test Strategy?

A test strategy is the top-level artifact that gives visibility to how testing will be approached for a given project, as agreed to by the stakeholders, typically addressing:

  • Purpose / scope
  • Quality criteria
  • Assumptions & constraints
  • Test approach
  • Inclusions / exclusions
  • Types of testing
  • Issues & risks
    Test Lead: “I don’t believe this. We are not going to have any time to test properly!”
Colleague: “Your new project?”    
    Test Lead: “Yes. My test strategy needs 3x what testing is now scheduled for. I can maybe cut a few things…I know I can’t just ask for more time or people…but… Oh, it’s so frustrating.”
Colleague: “Sounds like you have a challenge for sure. I agree you can’t just say you won’t have time to test everything. Have you included different test techniques and minimized their coverage overlap?”    
    Test Lead: “Yes. I can look at it again and maybe thin it down a bit more. But I don’t know if I can get it to fit. I need the PM to understand we can’t just not test some things.”

Quality Criteria and Acceptance

Quality criteria are those things that must be true for acceptance to be possible. They can be derived from risks, scope, and organization standards (“quality bar”)

How do we know, or have confidence, that the quality criteria have been met for a given release? And, is 80% confident “good enough”? 90%? 95%? 99.99999%?

Colleague: “Why do we test at all?”    
    Test Lead: “To find all the bugs, of course.” <smiling>
Colleague: “What?” <laughing>
   
    Test Lead: “I know. When have we ever been able to do that, right?”
Colleague: “Yes. I like to think about it in terms of risk mitigation. This lets me propose options or choices for examination.”    

Testing Objective/Motivation/Priorities

As testing is virtually always constrained with not enough time, not enough people, not enough infrastructure, etc it is vital that the testing effort is able to prioritize what is critical, and what is not, so the scarce resources can be applied in the most valuable manner possible.

One of the crucial components to being “smart” in a constrained situation is to be able to correctly decide or select which things are “must-haves”, which are “should-haves”, and which are “nice-to-haves”.

Having a mission statement, such as the following, can help keep your focus on what is important and reveal what is less-so, and give visibility of the same to the rest of the organization:

“The mission of the project’s Test Team is to undertake such test-related activities within the constraints provided to them by the organization to maximize mitigation of the likelihood of a failure of the software system from occurring after deployment that would impact the business.” *

* This example is for an embedded test team on an enterprise scale project. Tailor your own mission statement to fit your team’s scope of responsibility for quality within your project and organization.

Colleague: “Are your budget and schedule already firmly set?”    
    Test Lead: “Pretty much I think. I could try to work with the PM on those though. I have before, when I had a good case.”
Colleague: “That’s good. Then you can put together 3-4 options to show the tradeoffs of sticking with the current schedule versus shaking things up a bit.”
   

Finding Your Best Option

In developing your test strategy, you might use a table like the following to compare the different flavours or options that you identify:

Comparison Criteria Option A Option B
Cost (during project)
Coverage (breadth & depth of testing)
Schedule required (elapsed time)
Resources required (avg / max burn rate)
Future re-use of test assets

Tailor the above list of comparison criteria to your needs, perhaps including others like: test auditability, repeatability of test steps, early involvement/feedback, manual vs. automated tests ratio, test preparation vs. test execution effort ratio, Total Cost of Quality (slide 5), etc.

Note: In creating your options, you are really trying to find the “best” option. Therefore, any initial option can be “looted” for pieces to be merged with others on the way to creating that final, agreed approach.

    Test Lead: “Thanks for letting me vent a bit. Now I’ve got to go crunch some numbers.”
Colleague: “No problem. Good luck and let me know if you want to do a walkthrough before presenting.”
   

Test Planning Framework

Consider the following steps in your test planning process:

  1. Review the project scope, delivery schedule, and constraints
  2. Determine if/how testing activities are able to assess and/or mitigate the risks (to quality, the technical solution, and the project itself)
  3. Capture the needs of the stakeholders in the testing effort
  4. Confirm the test artifacts required and the level of formality
  5. Identify available team members, tools, environments, etc.
  6. Outline reasonable options for the project’s test strategy
  7. Support the options with test effort estimates and test project schedule/resourcing data
  8. Present test strategy options for review and input by project team and stakeholders
  9. Iterate on feedback and converge to the selected/agreed approach

Gaining and maintaining this agreement between the stakeholders for the duration of the project (as things progress and change) will enable testing activities to be appropriately prioritized so as to successfully test the right things at the right time within the project constraints.

Summary

Test Strategy Co-dependenceThe Test Strategy, the Test Estimate, and the Test Project Plan are all co-dependent on each other which causes a cyclical dependency between the three artifacts. ie: a change in any one of the three artifacts should result in a review and possible update of the others.

And, when these three artifacts are driven by risk, the activities captured will be strategically prioritized, making it straightforward to decide what tasks could be most confidently dropped if there was a sudden need to do so.

For related reading, check out these articles:

Avatar

About Trevor Atkins

Trevor Atkins (@thinktesting) has been involved in 100’s of software projects over the last 20+ years and has a demonstrated track record of achieving rapid ROI for his customers and their business. Experienced in all project roles, Trevor’s primary focus has been on planning and execution of projects and improvement of the same, so as to optimize quality versus constraints for the business. LinkedIn Profile
This entry was posted in  All, Estimation for Testing, Planning for Quality, Test Planning & Strategy and tagged , , . Bookmark the permalink.