3 Amigos - a sequel

In agile, the 3 Amigos is a collaboration event that takes place between QA, Product and Engineering right before the start of software development. The purpose is develop a shared understanding over what is being built, and what will be tested In this new version, the 3 Amigos still keeps three core roles. But these are now Product Owner, Software Engineer and Support.

Why the change you ask?

Modern Engineering Practises

Many companies I work with no longer have the role of QA as in the traditional sense. In these companies, software engineers perform the activity of software testing, often both automated and exploratory. Having a QA in a meeting where no QA exists doesn’t make sense.

But that’s not the only reason for the shift.

It’s time to acknowledge and formalise the need for operations in these discussions.

Whole Team Quality

Those walking the talk on whole team quality, realise we cannot ignore the impact of ops and customer support to this discussion. Elisabeth Hendrickson’s blog post on recovery over perfection is now 6 years old…In this post post, Elisabeth talks to how contemporary products demand we focus not only on great software but also the ability to recover from the inevitable failure that will occur. This ideology has a significant impact on how we think about quality and software testing. Instead of thinking only of testing, we in the quality space must be thinking about recovery. We need to be actively having conversations on how we can include operations into all discussion on bug prevention, detection and recovery.

We need to be actively having conversations on how we can include operations into all discussion on bug prevention, detection and recovery.

This is not necessarily an easy transition (what significant shifts are?). For ‘whole team quality’ to work well, requires a conscious effort to collaborate with individuals and roles outside of our comfort zones. I’m aware this is not an easy task, especially when people are under time pressure. Going the extra mile and having a conversation with someone outside of your remit is bloody hard when you working at full tilt to get product out the door. Yet, it’s vital if we are serious about improving quality. Reframing the 3 Amigos is one formal way we can begin the shift the conversation to be more inclusive of operations and customer support.

Software as a Service

There’s a few reasons this is now more important than ever. The complexity of our systems today, has put to bed the myth of delivering perfect software. We formally recognise, its impossible to deliver 100% bug free software. In addition, we no longer deliver product, instead we’re delivering an ongoing service (Jeff Sussna). We’re deploying frequently, in with a trend to smaller and smaller batches, and, our businesses are loving it. The fear of a release is dissipating with the knowledge that fixes and mind changes can be rapidly turned around.

If we decide let go the fear of releasing imperfect software (conceptually and emotionally), we need to make sure our operations and support are able to effectively hand the flow on effect. We need to be able to recover from bugs in production quickly, perhaps even before our consumers are aware the bug exists. This means we have to be able to detect and identify the root cause of a problem quickly. We’re talking observability, being able to see inside a system without having to add anything.

Observability like testability needs to be designed in. It requires distributed tracing & appropriate infrastructure to facilitate debugging in production. All this requires we collaborate with operations better. Of course, devops provides us with a partial solution to this. The whole purpose of devops is to improve collaboration and share responsibilities for both success and failure.

The primary characteristic of DevOps culture is increased collaboration Rouan Wilsenach (MartinFowler.com)

But this collaboration does not include the product owner. To develop a complete shared understanding of an upcoming feature, the product owner is key. They hold the vision and the why of the feature. They are the definers of success. Without them existing we wouldn’t be developing software in the first place. It’s time to break down this silo and build a conversation.

New 3 Amigos

The new 3 Amigos facilitates this collaboration.

A conversation prior to software development between 3 key participants, the product owner, the software engineer and the ops engineer needs to take place that seeks to prevent, detect and recover bugs.

Bug Prevention

The prevention of bugs happens in the same way as a traditional 3 Amigos. Missed scope is identified through the use of clarifying questions. Questions like, “where else might this feature be used”, “who else might need access to this feature”?

Technical risk is explored too. “Have we thought about security, usability or accessibility?”, “How many people are we expecting to consume this feature, and how will this impact our reliability?” “If a feature is not usable, what will be the impact on customer support, and are we ok with that?”

Bug Detection

Bugs are detected through testing the software. Here we want to come up with an agreed testing strategy that all agree on. Here you can use the 4W’s of software testing to guide. What will we test? Who will perform it? Where will it be done? And when will it happen? Don’t forget to ask about testability too. Ask: “How will we test this”? You might generate an interesting discussion, even change the where and when a test will be executed.

Bug Recovery

And finally we ask about recovery of any bugs that slip through the net. How will we know in production if a problem occurs? How will we quickly recover in production from failure? What measurements are important to us? What tracing is required that enables us to detect such a problem in production? How will we deploy to mitigate the risk?

What about the Testers?

If a software tester or a QA exists within the organisation, then they play a vital role of facilitation and providing a structure around which these 3 Amigos can collaborate. But if there is no software tester, a quality coach takes on this role. It’s worth repeating though, this is not a role of ownership but of facilitation.

Instead of directly asking the questions, you coach others to be able to ask these questions. You create the structure and event and help people through the process.

Challenges with the new 3 Amigos

All this doesn’t come without difficulty and challenge. One being a shared language and shared tooling. Put an ops person and a product person in the same room, and though they may have the same intent (a success feature),the terminology these roles use are different. And of course, there’s the question of priority. What matters most to a product owner, may not be top priority for an ops person. Plus, how and what we measure differs in software development and operations (Mark Tomlinson talks to this on the latest quality roadshow podcast).

Tooling between software development and operations can be different requiring different know how when it comes to debugging in development and operations . Those barriers are slowly coming down. For instance, Honeycomb encourages the use of their tool in all environments local, shared and production, thus providing a shared context and shared understanding.

I feel this is only the beginning of a trend. There is so much opportunity to expand on this collaborative event. For example, imagine a discussion on measuring feature success with the ops team in the room? Or asking questions on the impact of scaling a feature? I plan to expand on this in a second post.

And that’s the crux of it. The new 3 Amigos. Let me know in the comments what you think and please give it a try in your company and let me know how it goes.