From Bulldozer to Scalpel

I've worked on a number of large transformation programs. One of the hardest challenges is aligning a change to a high-level metric. How does rolling out a test-automation tool affect defect leakage? How does introducing a quality coach change delivery outcomes?

These questions are notoriously hard to answer. The truth is, what looks like causation is often just correlation. Sure, your metric moved — but perhaps not solely because of the initiative you rolled out. What went unmeasured was the new leadership or the org restructure. And because they weren't measured, their impact goes unnoticed. Any high-level metric shifts for a range of reasons, and many of them sit outside your radar.

Engineers treat metrics with extreme wariness. We know metrics are proxies for something you can't count. We measure test coverage because "quality" is subjective and constantly changes. Point a program at a number, and teams will move the number, whether or not anything improved.

None of that survives contact with a budget meeting. I've been on both sides of this, in consulting and in salaried roles, and here's what I've learned: arguing why metrics are poor measures wins the argument, rarely the money. "It's complicated, metrics are proxies" doesn't fund the work. Being able to show that something the company cares about shifted does. I know it sucks, but it's the reality.

So what's left is pragmatic. Use several metrics so no single one drives behaviour, accept the ones you're handed, and improve what you can. No answer is optimal. For me, pragmatism beats ideology. This leaves me with a slightly sour taste in my mouth. I know it's the right choice to make, but I'm not exactly comfortable with it.

Measure the gap, not the goal

The way out isn't a better metric or a better argument. It's to look one level down, at the failure modes underneath the metric.

Instead of "did this initiative move flow efficiency," ask "did this initiative close the code-review wait that a team demonstrably has?" It's a clean question: the wait dropped, or it didn't.

And review wait-time is a known component of flow efficiency. So the line of sight runs the other way: flow efficiency, down to the wait state that feeds it, to the failure mode, to the intervention aimed at it.

This is the Signal Engine

This is the engine I described in the last post, built on the design patterns from the first. It starts from the idea that failure will always exist, so we should learn from it rather than treat it as a threat. Here, the failure data does one more job: it drives the change.

It reads the engineering and ops data, finds the failure modes a team actually has, matches an intervention to one, and measures whether the gap closed.

Fit, not mandate

Transformation programs are expensive and unwieldy, perhaps why consultants love them. They're driven from the top down and often have little relevance to the change a team actually needs. They're the cod-liver oil of enterprise — you take it because it's good for you. Every team is told to adopt the same initiative regardless of context, which is impossible; you can't apply contract testing to systems that have no APIs.

However, that doesn't mean they're ineffective. In organisations where tickets are the currency, a mandated task eventually gets done. It's using a bulldozer to hammer in a nail — clumsy, but the nail goes in. What it costs you is the team, who know better than anyone what the real problem is, and would rather spend the effort elsewhere.

That real problem the team knows about? It's a failure mode they feel the impact of every day. The reason might be obvious — long waits on peer review. Or opaque — stories sliced too big. These are the failures the engine reads in the data. Because the gap it measures is the team's own, so is the fix.

A blanket mandate ships contract testing to every team — including the ones with no API to test. The Signal Engine matches instead: contract testing where the gap is interface drift, something else where it isn't. Same catalogue, different pick, because the gap is different.

That's it. The failure mode you use to prove impact at the budget meeting is the same one that makes the fix fit the team. One buys you the line of sight; the other buys you the relevance. A top-down initiative stops being a bulldozer dropped on every team and becomes a scalpel aimed at the one thing that team needs cut.