Product Excellence - Continuous Quality from Discovery to Support
Overview
Introducing Product Excellence, a new framework that creates a single view of quality across discovery and delivery.
Delivering small batches of customer value in isolation comes with a cost. As a strategy for risk reduction, it wins hands down. It can come up short as a strategy for delivering a quality product. This is because deconstruction often means we lose sight of the big picture, which hampers our ability to evaluate and comprehend overall product quality.
Silos are great for focus but awful for quality. Threats to quality often grow in the space between silos. Here, miscommunication and ‘zero touch’ communication thrive, resulting in missed requirements, hidden dependencies and poor estimation. To combat this, we must discover ways to bridge the gaps and begin a conversation on quality grounded in shared understanding, critical thinking and risk-based analysis. In this way, we avoid Chinese Whispers where quality for our customers becomes a miserable facsimile of our original intent.
Discovery and Delivery….and support
Continuous Discovery Habits by Theresa Torres describes continuous discovery as consisting of discovery and delivery as separate yet symbiotic activities.
From a quality perspective, the product doesn’t end at delivery; it extends into technical operations and support. Nowhere is this more true than in a SaaS product. For SaaS, it’s more than discovery and delivery; it's about maintaining and supporting our delivery. In SaaS products, we want to "figure out what to build", "then building it" and "then maintain and support it". We might even have to consider decommissioning and the data migration, security, and safety and potential job losses that go along with that.
In this context, quality is building the right product, building the product right, and maintaining and supporting it well. Depending on your focus, different perceptions of quality exist throughout our product lifecycle. A product manager may view quality through the lens of building the right product. An engineer focuses on building the product right, and technical operations look to support and maintain that product as it continues to serve our customers.
Quality is value to some person
Optimising for quality is useful and problematic. Deconstructing quality into different dimensions optimises our tests for the required information. An engineer can focus on scalability, while a designer can focus on accessibility. The result is that our proxy for quality in accessibility will be different to that for scalability.
But deconstruction can cause us to lose sight of the big picture. The big picture is that quality is ultimately a relationship between the customer and the product. When we lose sight of this, we frame our experiments to suit our optimisation. It results in teams believing they have a common understanding of quality but have wildly disparate views.
We want a model that allows us to generate a shared understanding of quality and how we will test for it and report on it.
Product Excellence
The Product Excellence model provides a holistic view of quality across discovery, delivery and support. By developing a shared understanding of quality and maintaining that perspective across all facets of the product lifecycle, we can prioritise quality to maximise customer value.
The Product Excellence model is built upon Gabrielle Benefield’s Mobius Loop, where the product lifecycle is more cyclical than linear.
The model resembles Janet Gregory’s and Lisa Crispin’s holistic testing model. However, the focus is broader across all elements of product discovery, and the conversation begins with a shared understanding of customer value.
We must think holistically about quality rather than distinct product lifecycle activities. Right now, we don’t do this. We optimise our testing in the same way we optimise our roles and activities. We don’t have a common language on what quality means to us. Consequently, our testing is framed and measured in different ways and almost different languages. So even though we all passionately want to deliver quality to our customers, we rarely discuss that.
Customer Value
We must find commonality and a shared goal to help us begin that conversation about quality. Once we have that, we develop and sustain a cohesive view of quality across all activities. I believe the unifier is customer value. A business exists to deliver and profit from customer value. Product teams exist to ideate, build and maintain that customer value. It both unifies us and allows us to suitably optimise for differences. As you see below, each role within a product team focuses on a slightly different element of that customer value.
Testing is everywhere!
I once attended a combined discovery and delivery workshop. We mapped out all the activities performed across discovery, delivery and ops. It turns out we perform testing in a lot of places.
Here are some testing activities identified:
- Hypothesis testing
- Assumptions testing
- Unit Testing
- Integration Testing
- Usability Testing
- Acceptance Testing
- Performance Testing
- A/B Testing
- Canary Releases
- Testing in Production
The purpose and nature of the test may differ, but it seems to hold that in tests we trust! We need to unify these efforts with a shared understanding of customer value. Once we have that, we can begin framing our experiments this way. We can allow for proper estimation to conduct these experiments, and we can make sure everyone in the product team can understand the state of quality at any time.
Good Test Design
I believe we instinctively test for customer value. We test to confirm the existence of customer value or at least a proxy of that value. We test for the “known knowns”.
We sometimes neglect to test for failure, the “unknown unknowns”. Unless we train or are hired for that purpose, the instinct is to test that it confirms our hypothesis and not realise that we should also design tests with failure in mind.
These “unknown unknowns” catch us out, surprise us, and turn our calculated bet into a messy gamble.
Critical Thinking
Critical Thinking is the antidote to these nasty surprises.
Critical thinking helps us to maximise our efforts in testing. Instead of trial and error, we become intentional about what and how we test.
When we view quality as a whole, we can begin to think critically across every portion of the product lifecycle. For example, we can ask: what does “build the right product” mean? How would we know we’re building the “right” product? What does success look like for that? And how important is the information in the big scheme of things? If our tests fail, what decisions will we make?
We must have a collective view of quality and understand the subtle nuances each role offers to that perspective. We must understand how to test for it and how it will inform our decision-making process. Mindlessly ‘testing everything and trying to automate all the things’ is a pretty low bar for a testing strategy. However, framing our testing with a shared understanding of quality allows us to maximise the value of our testing.
Critical Thinking in Test Design
We instinctively test for customer value. Our tests confirm that the chosen proxy of customer value suggests that customer value exists.
Unfortunately, the process of creation is subject to confirmation bias. As a result, when we create something, we lean to confirmatory testing and omit to test for failure. There are a couple of ways to mitigate this risk. You can train product teams on good test design or have a specific role that questions and evaluates quality.
Regardless, thinking critically through your testing strategy is a good idea. The information that results allows us to make informed decisions, turning our gamble into a calculated bet.
Risk Focuses our work
If that sounds like a lot of testing, you are right. In many contexts, we’re unwilling to invest in that rigour, instead trading our quality off against speed of delivery and cost. There is an alternative approach. Using risk as a focusing technique, we can identify what is critical and must not fail and focus our testing efforts around that.
Risk is anything that threatens the quality of the product. Risk helps us quantify failure by assessing its impact on business value and the likelihood of failure. Designing our tests for potential failure helps us focus the testing effort on “must do”. Instead of attempting to test everything, we focus on the testing effort where failure may have its most significant impact and is most likely to occur.
Quality Professionals
I believe quality professionals (QP) are in the ideal position to be the golden thread that links product, delivery and support, as in their existing role, they need extensive knowledge of business value and technical complexity.
QPs are adept at using risk to evaluate the quality of the product quickly, so much so that people assume it's some superpower. It's not. It’s years of tacit knowledge acquired through discovering defects in products.
Quality professionals are like navigators on a rally team advising the product team to speed up when the risk is low or slow down when it's high.
Quality professionals are adept in designing, executing, and reporting on experiments. That is the essence of software testing. QPs understand the importance of good test design and solid execution of tests that offer consistent and reliable results.
Next time you think about product quality, pull your quality professional into the conversation. Include them in prototyping and customer experience sessions so they build up knowledge of what customer value looks like. As ideation moves into delivery, they become the product champion, keeping testing conversations focused on customer value. They will initiate discussion on quality and how to prevent, detect and recover from failure. They become a conduit of information, helping delivery with domain context and providing progress on the state of quality to delivery.
Let them test in production behind a feature flag. Allow them tools such as Grafana, Prometheus and Honeycomb that allow them to analyse customer interactions in real-time.
Along with technical success, allow them to feed this information into discovery and ideation.
Your customers will thank you for it.
References
Teresa Torres - Continuous Discovery Habits https://www.producttalk.org/2021/08/product-discovery/
Quality is value to some person was coined by Jerry Weinberg
Janet Gregory’s and Lisa Crispin's Holistic Model https://janetgregory.ca/testing-from-a-holistic-point-of-view/,
Tester as Navigator is an XP reference used by both Lisa Crispin & David Evans, respectively. Testing in Extreme Programming by Lisa Crispin & Tip House
Gabrielle Benefield Mobius Loop https://www.mobiusloop.com/
Ask the Book
Welcome! Ask anything from the Handbook.
Comments ()