Decoding the Dance: Customer Value Meets Quality

The voice of the customer

Even before I started in this industry, there's been an interplay between customer value and quality—Juran's 1962 2nd edition of the Handbook of Quality talks about the interplay between the two.

In software engineering, quality typically means 'product quality'. When quality professionals describe their role, it's often to be a 'voice of the customer' providing that perspective to the rest of the team.

Jerry Weinberg, the doyen of software testing¹, defined quality as "value to some person". Value and quality are intrinsically linked and sometimes mean the same thing.

The current elevation of product discovery in our software product lifecycle, particularly in SaaS, means we must explore this relationship again. How does an emphasis on innovation of customer value impact our understanding of quality?

Discovering Customer Value

What exactly is product discovery, and how does it interact with product delivery^?

Product discovery is an exploration of the problem space to discover customer value. Product discovery investigates the problem space to identify hypotheses on potential customer value. We only know if these hypotheses are correct once we deliver that value to our customers. Product delivery is that medium. The process of designing, building, releasing and maintaining software allows us to know for sure if business value is delivered.

It's not unfamiliar territory for most of us. Many say quality is "building the right thing and building the thing right"³.

We've previously treated these two phases as entirely separate entities. We look at the quality of discovery independent of the quality of delivery. In SaaS, we can't do that because the success of product discovery is dependent on the speed of product delivery. In fact, for some, dev velocity is a proxy for innovation⁵.

Building the right thing over building it right

Failing to build the right thing is more important than failing to build the thing right. If we make a product that nobody wants, how perfectly it works is irrelevant. Building a desirable product means how it works will matter more.

That means if a tradeoff between the two is required, the ability to innovate customer value matters more than the ability to deliver a perfect product.

That tradeoff exists and is called "speed to market". Because customer value is constantly changing, morphing, and evolving, the ability to quickly iterate and test assumptions on customer value is critical.

This is not great news for quality in the delivery space, where time is already the enemy of quality^4. It could potentially impact the ability to test for product quality well.

Do we accept this tradeoff, or can we identify new ways to ensure quality that perhaps look beyond software testing?

SaaS value triangle

My definition of quality for SaaS models is:

Quality is how fast we deliver customer value, how stable and constant that customer value is in production⁶

In short: Quality = throughput + stability⁷ + availability

Like the iron triangle, these factors have a relationship:

1. The value of work relies on stability, availability and throughput.
2. A product team or quality professional can trade between the elements.
3. Changes in one quality characteristic necessitate changes in others to compensate, or value will suffer.

Reduce time and quality increases? Strange days indeed. I would argue that this definition of quality is closer to how senior engineering stakeholders perceive it.

Time will tell if this model holds. For now, it's more speculation than theory. My question to you all is: How might your thoughts on quality change if this is true?

Diversity in your Quality Strategy

The unfortunate reality is that software testing is a time-intensive activity. Designing, creating, executing, reporting, and maintaining software testing artifacts takes time.

We do these time-consuming activities until companies have seen their value. Software testing allows companies to reduce the risk of failure in production. Suppose another method of reducing the risk of failure in production were available, one that was less time-intensive yet allowed for a similar state of quality. I do not doubt that companies would explore that.

Instead of depending on software testing as the only means to identify the quality state, we look at other techniques and approaches. A diverse quality strategy not only reduces dependency on software testing, but as a strategy in itself, it reduces the risk of overreliance on one approach.

Check out my Prevent, Detect, Recover article for ideas on creating a diverse quality strategy.

New Quality Attributes

And throwing this in, if you agree that the definition of quality has changed, doesn't that mean we need to relook at the quality characteristics, too?

My potential new set of quality attributes is:

• stability⁷
• availability
• throughput
• usability
• accessibility
• observability
• qualtability
• security
• scalability
• serviceability⁸

In conclusion

I suggest we begin exploring different ways to think about quality. Let's take a step back and critique our existing assumptions on quality. If we question what we accept as truth in testing, we may identify new ways of achieving quality products that our customers value.

References

¹ Read Perfect Software by Jerry an excellent book on software testing

² Teresa Torres's excellent book "Continuous Discovery Habits" and her website are a good place to start understanding the product discovery space.

³ My article on product excellence provides an overview of this

⁴At least, that's what the iron triangle tells us.

⁵ Read this article by Blameless on how a proxy of discovery is the speed of delivery.

⁶ This definition is heavily influenced by Accelerate and the 2018 State of Devops paper.

⁷ Stability is another word for functional correctness and is measured by MTTR and Change Failure Rate

⁸ Serviceability is how easy or hard it is for others to use your service. For example, is it easy to pick up and use an API?