Methods of uncovering defects have for the most part stayed the same even with great advancements in process and development tools. One thing that has not stayed the same is the amount of time we have to uncover these defects. With this time constraint how can we uncover the high value defects which could be costly to our organizations? What shift in test technique do we need in order to tackle this time constraint and not fail fast in a horrible way?
A Quality Foundation
In order to detect high value defects we cannot have software which is full of low value trivial defects. When we do not have a quality foundation or reasonable level of quality before testing begins the following occurs:
- Testers stops testing to log or inform a developer of a trivial defect they have uncovered. (Testers need to be testing to uncover high value defects.)
- Developers stops developing in order to learn about trivial defects.
- If a decision to fix this trivial defect goes forward most often times the developer is out of the context of this work. It will take them more time to re-learn or re-gain context in order to apply a fix.
- Trivial fixes can cause more defects.
- If you have a quality process in place after this trivial fix is made there is cost associated with it. Continuous Integration systems – build and test jobs along with developer code reviews take time.
- Finally and most importantly, because you are spending so much time uncovering/fixing trivial issues you can never reach the deeper high value defects.
Building a Quality Foundation
In order to avoid the negative points outlined above we must ensure a baseline of quality is always maintained. Again without this we will be lost uncovering, triaging and fixing low value defects unable to expose the defects which are the most costly. We can build a quality foundation using the following techniques:
Automation Tools (Checks)
- Automation is a great way to maintain a consistent level of quality throughout the development cycle. Build on this foundation as your developers develop. With new features add more coverage.
Manual Test Review
- Code reviews are standard practice on most development teams. Taking this concept a step further, why not provide a test review? This can be a small manual test check for a feature before the code is checked in for further in-depth testing. Note: not all development changes require this manual check but if you find you are having a lot of trivial findings you may want to try this on your team.
It’s worth highlighting that automation tools are well suited for creating a quality foundation however, many of the high value defects we wish to flush out will not in my experience be uncovered by automation alone. This is because automation tools check/verify software and do not test software. Testing software requires a human to think, it is not simply checking that the correct screen appears after tapping a button.
Use automation tools for what they do best ensuring a baseline quality foundation continuously at high speed. Don’t expect automation tools to think and therefore have the capability to find high value defects.
Gain Context
Now that our quality foundation is set, what knowledge do we need in order to maximize our ability to uncover high value defects? In order to make our testing more valuable we need to gain context about the software we are going to test. The following activities can help you gain context:
- Understand the Feature – This seems trivial but have an understanding why a feature is being added to your software. Also understand what type of user will use this feature. This can help you understand how this feature should be properly exposed in your software. High value defects are not always crashes, a poorly implemented feature is also a high value defect/problem. These findings also expose opportunities to make features work in simpler/better ways. It’s worth noting understanding a feature should start as early as possible. Ideally when user stories are being created.
- Development Tours – When a developer finishes implementing a feature/bug fix, the tester can pair up with them to get a tour of the feature or bug fix. These tours can help testers gain key insights on how a feature was implemented. What problem areas are there and what other areas of the code needed changing to implement the feature.
- User Feedback – No matter how good you think you have implemented and tested features you won’t get it 100% right. If you have access to user feedback you should make it a habit to check this feedback every day. Gaining a deeper understanding of pain points in your software from a user’s perspective, can help you when testing future features.
- Production Logs – Similar to user feedback reviewing, crash logs from production can help you understand what areas of your software are buggy. When testing you might take more time in these error prone areas. The entire development team should know about these areas as well. As a tester you should share this information.
- Competitive Analysis – Understand your competitors strengths and faults. Don’t repeat mistakes they have already made when implementing features.
Pre-test Plan
Ok in no way am I suggesting you drop everything and create a large test plan. My experience tells me this practice in most ways is a waste of time. What I am suggesting is spending 5 minutes figuring out the following:
- What states can the software be in when interacting with this new feature
- What inputs can be used to exercise this new feature
- How usable/accessible is this feature in our software
Think about the testing you will perform. I find diving into testing without first thinking about the testing you will perform can be a bit of a blind strategy. An experienced tester will still find defects without this approach, but for me I find this helps frame my testing.
Testing
Your quality foundation is set, you have gained context around what you will be testing and you have a rough idea how you will approach your testing. You are now ready to test and are in a position to flush out high value defects.
A lot of what is written in this article is already done by great testers in our industry. I wrote this article in an attempt to understand what I do in order to find defects. I believe the exercise of understanding what makes you a great tester is a worthwhile one. So when you have time go through this same exercise and you may just uncover some great ideas around test. Please share these ideas.
Now go uncover high value defects!
