Let’s admit it. Projects that deliver enterprise solutions have barriers to being agile, especially those that are based on purchased packages (commercial off-the-shelf packages, or COTS). In my experience I’ve wrestled with at least two of those – the technical barrier of test automation capability and the organizational barrier of having a single Product Owner that truly speaks for the whole of the enterprise.
In this post, ‘user acceptance testing’ (UAT) represents that last-gasp critical-path test activity that enterprise projects typically run just before going live. It is generally an entry condition for migrating the solution into the production environments. The theory is generally that the ‘business unit’ needs to understand and approve of what they will be using post go-live and that this is the last time to check that the combined new/revised/existing business processes are supported by the new solution, and vice-versa as the case may be.
The ‘test last’ bit and acceptance not being done during the delivery sprint(s) doesn’t feel ‘agile’. So then how could user acceptance testing be the gateway to agile for enterprise solution teams besides the general observation that a UAT window in an enterprise project is usually 2-4 weeks, that is, the same duration you might expect of a typical scrum project.
Gateway Characteristic #1: Whole-team Approach
Briefly, a UAT phase run in a critical-path manner is an ‘all hands on deck’ phase. The project or test manager usually rallies the team around the UAT cause in some sort of a launch meeting, where the people that are doing the testing meet the people that are supporting them doing the testing and everyone in between. Everyone participates in this phase – trainers, the business units that will receive/use the new solution, IT personnel that will support the solution post go-live, etc.
A wise test manager will in fact plan UAT this way as well by hosting workshops with the business to identify and otherwise order the business process scenarios that would be in-scope of the UAT. To avoid creating test scripts that repeat the content of either training material or business process descriptions, the intent of these workshops is to identify the testing tasks/activities that need to be done and in what order they should be done in for maximum effect.
Does that sound a bit like sprint planning to you?
The whole team also comes into play in the problem/issue workflow, where there is generally a daily triage of discovered problems that need to be described, classified for severity, and assigned to a resolution provider. Generally everyone participates in these triage sessions for cross-communications purposes. The problems are not to be resolved in this meeting, just raised, classified and then picked up by someone that can do something about it.
Could this meeting be extended by 15 minutes so that everyone also mentions what they tested yesterday and what they will test today, that is, in addition to what problems/barriers/issues they are running into?
Last point in this section – most of the companies that I’ve managed UAT on behalf of establish a “UAT testing room” that is a co-opted training facility or meeting room so that all the business testers can be co-located when they are actively testing. Hmmm. Sounds like what we generally arrange for an agile team.
Gateway Characteristic #2: Intensive Business Involvement and Leadership
It is common to warrant that business involvement and leadership exists by getting the UAT plan signed-off before the phase and a report of the results signed-off afterwards. That might be necessary, but it’s not sufficient for any UAT. It is critical that there be business people actively testing – in fact they are the only ones that can perform the testing and acceptance according to the generally-accepted governance umbrella that the UAT operates under. Since there are always problems detected in UAT, the business leadership needs to pay attention and get involved when their people might get discouraged because of what they find or believe they are finding, as the case may be.
It’s a nervous time for the business people involved in testing and for their leadership to dump all the responsibility and accountability onto them for making the acceptance decision is counter-productive and harmful. First, these aren’t professional testers – they are people from the business units seconded to do this testing and second – they might not even like testing and third – they might still have to do their regular jobs during the testing. The right mix of leadership from the business unit and from the project team implementing the solution is required to support these individuals.
The common characteristic is that both agile projects and UAT require that business leadership touch to be successful.
Gateway Characteristic #3: There is a Backlog
Most of the time – but I suppose not all the time – there is a list of problems/defects that were discovered in earlier test phases that will be resolved at some point during UAT. Those problems are not severe enough to have prevented UAT from starting, but at the same time, they do still need to be resolved. In addition, the UAT test plan lists business scenarios that need to be run over the course of UAT. These two lists are combined and generally otherwise ordered so that the time spent in UAT is efficient and effective.
This is a backlog and I’ve seen many teams successfully use burn-down/burn-up charts to communicate their test progress. Information radiators such as dashboards are commonly posted to either project room walls, project repositories, or both. Discovered problems/issues are incorporated into the backlog using a process similar to the way that an agile team deals with problems/issues discovered within a sprint.
Gateway Characteristic #4: The Backlog is Ordered
When a problem/issue is discovered and goes through triage, it’s been ordered along with everything else that has to be done within the planned UAT window. The agile characteristic of this is that everyone has participated in the prioritization – business and technical personnel together make these decisions.
Gateway Characteristic #5: Business Acceptance of Detected Problems Happens Within the Phase
Detected problems go through their entire life cycle within the UAT window. They are identified, triaged, fixed, re-tested, resolved and closed before the phase ends. If not, then the problem is serious enough to either extend UAT or to stop it outright and halt the go-live. Consequently, there is enormous effort put in to make sure that there are adequate resources on call to get any problem resolved – training materials, software, business process, environment – any root cause whatsoever will generally get all the attention it needs to be taken care of.
As a gateway characteristic, this is exactly what the business people involved in an agile sprint are asked to do – identify what they need, compare what they are given to what they need, back-and-forth, reach consensus on what the final product is, accept it and move on to the next item. Business units therefore DO have experience with this kind of acceptance regimen if they’ve been through a UAT.
Gateway characteristic #6: The Result is a Potentially-shippable Version of the Solution
At the end of both a sprint and a UAT phase, there is a potentially-shippable version of the solution.
Concluding Thought Experiment
Generally a COTS package is selected after some period of analysis and review – a gap analysis per se – and then the implementation is planned based on closing those gaps. What if the needs and gaps were planned to be closed in a series of say 10 UAT-like phases? The two highest barriers remain the test automation problem – since many of these products do not automate well – and the difficulty filling the Product Owner role in large projects.