Release Criteria – What Is Your ‘Quality Bar’?

In a previous article, I discussed managing risk with quality gates (“None Shall Pass…unless? Managing Risk with Quality Gates”). Such gates and their expectations facilitate tracing issue root cause and examining which preventative measures failed or need to be enhanced so as to continue to lower costs associated with poor quality.

But that last gate is there to enforce a level of standard, or ‘quality bar’, for what is ultimately to be published, deployed, or otherwise seen outside of the project team itself.  To guard that gate, we need “release criteria”.

Assuming that you won’t be checking against these criteria unless you have passed all prior gates, release criteria are the last few things that need to be ‘just so’ to make your release a success.  Or more bluntly, meeting these last criteria is required to allow you to make a release at all.

Go/No-Go – A Rare Binary Decision

What is important at the point of making a release?  As a team, we need to be sure that both the technical and business risks associated with the value we intend to provide have been mitigated to acceptable levels given the purpose or intended usage of the release.

How do we evaluate whether something has been ‘mitigated’?  As a team, we need to describe a set of criteria in a quantifiable way such that they can be confirmed or not by a simple ‘yes’ or ‘no’.

Black, White, and a dab of Grey

To keep the focus clear and create the best chance for a release to make it out the door without compromising our standards, we can separate release criteria into two categories; what MUST be achieved before a release can happen and what is EXPECTED to be achieved.

The following are the minimum criteria that MUST be met prior to a given build being released. If these criteria are not met, the build will not be approved for release and corrective action must be undertaken and given the highest priority.

EXAMPLE: Minimum Release Criteria
All unit tests or peer reviews of code changes must be documented and pass
All “Fix Today” priority issues raised during testing of the series of builds leading up to the release or found during the release process are confirmed resolved (i.e. none are outstanding) via testing.Here a “Fix Today” priority issue is defined as an issue that meets one or more of the following conditions:

  1. In the issue database, a given defect or change request has a “Fix Today” priority set prior to the delivery of the applicable build to QA/Test. Ie: the issue is already scheduled for the build to be released
  2. A new defect that has been raised during testing of the build because of a critical failure of the system and no reasonable workaround exists
  3. A defect that is stated to be resolved in the issue database is discovered by testing to be not resolved and must be re-opened
  4. A new defect that has been raised because of a behaviour or functionality that was previously correct no longer working in the current build
  5. A new defect that has been raised because of a failure of an automated or manual smoke test that can be run successfully on a previous build
Every testable defect fix has been confirmed by testing to be implemented successfully
All acceptance tests must pass
The release notes will document if a particular change request or new feature is included in the build but has known open issues against it at the time of release

The following are the additional criteria that are EXPECTED, but not absolutely required to be met for a given build prior to it being approved for release.  If any of these criteria are not met, corrective action will be undertaken following a risk/impact assessment of the unmet criteria (eg: non-conformances will be logged as issues of varying priority).

This provides the project team the ability to react to other situations should they be determined to have a higher priority at the time.  However, it would be expected that corrective action would be undertaken in time for the following release.

EXAMPLE: Expected Release Criteria
Features that Product Managers/Customers have been promised are implemented
Every testable change request or new feature that has been implemented has been confirmed to be implemented successfully and confirmed as such through verification of documented expected capability and behaviours
No spelling mistakes in the screens within screens or documentation modified by change requests, new features, or fixed defects (e.g. typos or inconsistencies found in any other screen can be automatically added as a “Fix Later” issue)
All change requests, new features, or fixed defects implemented in the build are included in the release notes

Going Back for More

If we return to the idea of quality gates throughout the project lifecycle, we can add another dimension to release criteria.

Suppose we consider the promotion of a build from one environment to another a ‘release’ and, as such, establish criteria for this maturing of the system or code-base towards the final gate.

The following diagram provides an example of what this might look like at the highest level:

Release Criteria Diagram


If the project team has a ‘quality bar’ based on criteria that are practical and quantifiable, you will benefit from having a repeatable standard for quality in your releases.  Then challenge yourself to be even better.  Your customers will love it.

About Trevor Atkins

Trevor Atkins (@thinktesting) has been involved in 100’s of software projects over the last 20+ years and has a demonstrated track record of achieving rapid ROI for his customers and their business. Experienced in all project roles, Trevor’s primary focus has been on planning and execution of projects and improvement of the same, so as to optimize quality versus constraints for the business. LinkedIn Profile
This entry was posted in  All, Planning for Quality, Test Planning & Strategy and tagged , , , . Bookmark the permalink.