The goal of building software is to produce a product that will bring value to our customers. As software developers we have a reputation for finding and embracing new technologies. These new technologies, however, can only take us so far in ensuring the products we build are high quality and bring maximum value to our customers.
I fill the Quality Engineer role on my development team and I have been involved in software development for many years. What I now see in the software industry is a lack of customer focus and an over focus on technologies, processes that move the quality problem from a customer value problem to a mechanical technical one.
The thoughts written below have been floating around in my head nagging at me for the past couple of years and I wanted to share them with others to see if they resonate with anyone out there. I will walk through many of the challenges we face today when attempting to build quality software as well as possible solutions.
QUALITY
Quality is completely subjective. Ask each of your team members to describe what makes a piece of software high quality and you will get a number of different responses. These different perceptions of quality make it very difficult to come up with a strategy which will add up to building a high quality piece of software.
Lets us explore what our customer might see or perceive as quality vs what activities / behaviors / processes we use to attempt to ensure it.
Customer
- Solves their use case in a simple way which in turn brings them value
- Easy to navigate and find / discover features they care about
- Provides more value than the competitor
- Can continue their journey when they return to the software
- Familiar design patterns applied to customers platform
- How well software interacts with other software in which they are using
Development Team
- How fast we can deliver
- How many features can we deliver
- Test coverage numbers
- Checks / Gates integrated through a CI (Continuous Integration) pipe
- Using the latest technologies
- Metrics which internally we decide represent quality
As you can see, the customer’s perspective of quality and the activities / behaviors / processes we use in an attempt to ensure quality don’t line up. I believe this causes us to sometimes assert things that don’t always cover the entire quality problem.
We need to understand what quality means to our customers. They are our North Star let’s follow them.
DISTANCE FROM THE CUSTOMER
Multiple teams are often involved in building software. These teams can sometimes be assembled by layer, for example, front-end and back-end. Other times teams are focused very narrowly on one single widget on a website. The trouble here is we optimize for delivery speed without necessarily thinking enough about the customer or how these narrow teams will bring value or quality to the customer.
The distance from our customer here hurts our abilities to ensure our software meets their need and ultimately brings them value.
When we don’t keep our customers, our North Star close and constantly understand their needs we cannot bring full value to them. This distance impacts how we test and ultimately leads to lower quality software.
TESTING
Testing software has slowly become about what we already know. The software we build is constantly changing so simply repeating the same tests over and over doesn’t cut it. Thinking about testing needs to increase to understand what we don’t know. When we assert what we already know over and over we risk missing important discoveries that threaten our product quality.
Review the testing being performed by your teams and you will find activities that are quick to jump to locking / coding in automated assertions before deeply thinking about testing.
We have lost what matters the most in software testing, “the thinking”. This is not about automation being a bad thing. In fact automation and static assertions are fantastic at giving us a reliable quality baseline. Once an automation baseline is established this will allow your team to spend more time thinking and exploratory testing.
Thinking during test creation as well as observing test execution results will lead to a deeper understanding of how our software works. This continuous thinking often exposes not only functional issues but even deeper insights about your software. These uncovered insights are then turned into important future questions / tests.
This activity is what makes ok software turn into a magically solution for your customers. Move the quality bar up for your teams.
PEOPLE
What types of people are we hiring to ensure the software we build brings value to our customers? Are we hiring all the same type of people with very similar mindsets? Whoever fills this role must have a testing mindset – an ability to break up complex software and interrogate it, exposing risk early in our development process.
Strangely even when we hire QA or someone with the word “Quality” in their title we sometimes measure the value they bring to the company as the number of automated tests they have implemented not by how much risk they have uncovered.
I really feel this is not how we should measure a person who is testing on a software team. When testing, we are after exposing risk and also looking for opportunities to make our software better. Automation although very important is just one aspect of the required work.
If a team member regardless of their title understands your customer and consistently uncovers risk and opportunities in your software this person is valuable. You can call this person a QA, a tester or whatever you like. Take a look at what this person might be able to do to help make the team more effective at software testing and accomplishing your goals.
Don’t fall for the phrase software testers or QA is not scalable. We don’t need to hire large groups of QA for each layer of our software but I don’t want to hear testing or a person in charge of uncovering risk is not scalable. What’s not scalable is delivering software incrementally at high speed that does not provide value to your customers.
PROCESS
The shift to Agile although a good one has forced software testing to change. In some ways I feel this constrained test time is good. It should make what we deliver smaller and thus easier to ensure a high quality bar. The trouble is we often want to deliver so much so fast that there is not time for adequate testing. So what happens in this scenario?
We only focus on the basics of our features. We automate those and move on.
Even if you manage to release without any major functional issues chances are the software will only be at best functional. We should want more than this. We want great usability, performance and features that work intuitively. If there is not time for this we just keep delivering more and more average software.
ACCOUNTABILITY
If a major issue occurs in production that impacts your customers is anyone accountable? I’m not talking about walking over to someone’s desk and yelling about a missed issue but is anyone accountable? If no one takes direct responsibility for these quality mishaps it’s not long before the quality bar erodes lower and lower. Then you might hear things like:
It’s ok, our response time to fix things in production is extremely fast. The speed at which you can recover from problems is not the point. It’s great if you can recover quickly as sometimes teams will make mistakes but don’t lean on this mechanism over and over. Reward your team delivering quality out of the gate. Don’t reward for recovery from a problem which impacts your customers.
Your customer takes the brunt of this negative pattern. Your team needs to own your product and be accountable for issues that impact our customers. Talk about these mistakes and ask your teams how they might mitigate these types of issues in the future.
OWNERSHIP
Ownership can also be problematic when it comes to working towards building a quality product. Ideally, we want each member of the team to own quality. The real ask here is to have passion for what you are building.
When we simply say we all need to care about quality I believe this can wash away this goal. It gets watered down because no one person is accountable or has ownership.
Check the behavior of your teammates. If you’re lucky a few people will have passion and care about customer value. Others hide away from this ownership. Have a good look at who owns quality on your team and how that impacts the quality of your software.
REWARD
Reward and recognition is very important to ensure people remain motivated and appreciated for their efforts. Trouble is sometimes we over celebrate or reward internal achievements before the customer validates this as being a success. We have all seen it, the email chain floating around high-fiving teams internally. Meanwhile, customers are encountering quality problems in production. I know I might sound negative here but your customer will decide whether you have achieved success, not your internal team.
Let’s recognize our work internally but when our customers are happy then let’s really start celebrating success. This is what matters most.
MEASUREMENT / METRICS
Ah measurement. We all want to measure everything as it helps us make decisions. The problem with software quality is what do we measure. What tells us the right answer? As noted in an above point, quality is subjective so how can we get the right metrics?
Should we say we have 80% unit test coverage so we are high quality? Should we say we have executed x number of test cases so therefore we are high quality?
We can measure internal results but we really need to bring customer satisfaction with the software into the question. This is harder to do and that’s why I believe we stick to the number of automated tests to measure quality.
Find a way to measure how you are doing against your customer’s expectations. Figure out how to bring customers into your quality metrics.
DEATH BY CANARY
After the team has decided that enough testing has been done to ensure we are good to release to production we will often use a canary mechanism to slowly release or try out our new build against production users. This is a great mechanism to use as we do sometimes make small mistakes and knowing early and being able to mitigate these issues is a great idea.
With that being said if you find that every release you have is having issues then you are most likely abusing this mechanism. We shouldn’t use our valuable customers as crowd non-paid testers. Customers are real and when they encounter problems in production they may not come back.
Let’s not ignore the cost to our customers when we abuse this canary pattern. Yes we need it but if you have consecutive releases which need rolling back ask yourselves what might be the bigger issue happening in your development process.
DEATH BY PAPER CUT
We are often looking to the next feature because of the internal “Reward” I mentioned above but the fact is sometimes there are many small quality issues which need to be fixed. If we continually choose future work over fixing many of the existing small usability and functional issues we encounter quality debt. Fix these small customer problems. You will be surprised how much happier your customers will be.
Small fixes / tweaks to existing features are often more impactful than a future idea you have in your mind.
Most companies have the best of intentions in trying to build quality software that brings high value to their customers. Problem is they don’t live and breathe the customer. Let us explore some ideas below that may help us build better quality software.
SOLUTIONS
CONTINUOUS TESTING / EXPLORING
Just as early explorers looking for new lands did not sail the same routes each day it makes sense that if we want to discover potential defects or risks which impact our software we have to be continuously exploring. Specifically thinking about what might / could happen and its impact. The moment you stop this thinking your test approach is frozen. You cannot expect to discover everything about your software if you stop thinking about it.
Challenge the product your building. Go beyond scripts and use the product to understand where it might not make sense or how a small tweak could really improve your product. Understand that fixing three small usability issues could in fact be more valuable than your new shiny feature that may or may not be impactful for your customer.
AUTOMATION AS A BASELINE
Understand that the attributes of quality we can assess with tools do not make up the entire quality problem. They assist us in our journey but should not shut the door on human thinking about our customers and what we need to do to bring value to them.
Think of automation as a quality baseline mechanism. Something you can rely on to assert that the key pieces are working as expected. Once this baseline is established you can then think and explore your software. This exploring will uncover key findings that will help your product move beyond simply functional.
Don’t abuse patterns. For example, we have the test automation pyramid which explains that we should have most of our tests at the unit level then service level and then the UI level. From a technical side this makes a lot of sense however the customer ultimately interacts with the UI level when using our software. If we follow this pyramid as an overarching strategy we may be missing asserting how the product solves the customers problem at the most important layer – the layer they interact with.
REDUCE DISTANCE FROM THE CUSTOMER
The further you are from your customer the harder it is to understand what matters to them. A possible fix for this would be to schedule Bug Bashes. Bring as many team members from all layers and business functions as you can. When issues are uncovered during this bash you can understand as a group how your software works and where customers maybe encountering problems.
Meeting as a large group to discuss your product is so vital to success. When you do meet everyone should bring their voice and their angle to the discussion.
CUSTOMER FEEDBACK
If you do not have a channel which allows your customer to directly communicate with your development teams you are completely blind operating in a black box. You are basically in a storm and no North Star can be seen. Information shared by your customer is pure gold. It’s so special and amazing that your customers would be willing to share what pains them and is troubling their experiences. If you don’t have a feedback mechanism, stop everything and create one now.
After that the next challenge is getting your team members to read this feedback every day. You need this customer feedback to understand the difficulties and frustrations they are experiencing. With a better understanding of the customer’s problems you can make better decisions about fixes and what to build next.
BE YOUR OWN COMPANY
Don’t follow other companies test strategies. They have a different product with different customers. Understand there may not be a single gold solution to software testing. Grow and evolve your testing strategy over time, don’t lock in your strategy on day one.
Several large successful companies have very different test strategies. Why would we blindly follow these strategies? Review them but iterate on your own strategy. Measure what is working and delighting your customers. Lead don’t follow.
PEOPLE
Don’t hire people with the same mindsets and backgrounds. If you do, it could result in a very narrow view of software testing. A view which treats testing as a checkbox item which needs to be done but only just enough. We want to prioritize testing to ensure we are delivering top value to our customers.
Hire a dedicated tester for your team. I know agile does not mention having dedicated testers, but why should we blindly follow a process that negatively impacts our products and customers. Testers are inherently good at breaking software apart, exposing risk and guiding your developers. The way they think and expose problems with ease is a skill. At the same time, if you have testers ensure they are positioned where they will have the most impact. For example, don’t apply the same tester to dev ratio on every team you have. Position these testers in a layer that has the potential for the most problems. Doing so, could help guide your teams on what needs to be tested and how.
Finally, remember when hiring there is no substitute for passion. Passion trumps most other skills as most skills can be learned – passion cannot.
We all say we are customer first, hell at times we put this text into our mission statements but do our behaviors match up with this?
Great testing can help you deliver more value to your customer. If you have a test role on your team today make it a special role. Let this person not only expose risk but also guide your teams in making small tweaks which will enhance your software’s quality.
I really hope you’ve enjoyed the information I have gathered in this article. I have had these thoughts running around in my head for a long time but had trouble bundling them all up. In writing this article I have gained a clearer picture of what needs to be done to bring value to our customers through test.
We all say we are customer first but let’s actually be customer first!
