Last sprint of the quarter. Suddenly the team closes tickets like never before, velocity goes through the roof and the dashboard is green from end to end. In production, almost nothing changed.
And nobody set out to cheat. Stories got split in three, things that were halfway done got moved to "done", short tasks got picked before important ones. Each of those decisions, on its own, can be defended in a retro without blushing. All together they build a number that no longer says what it was supposed to say.
The economist Charles Goodhart saw it in the seventies, watching how monetary policy indicators behaved once the central bank started using them as targets. Years later the anthropologist Marilyn Strathern summed it up in the sentence that circulates today as Goodhart's law: when a measure becomes a target, it ceases to be a good measure.
I've never set foot in a central bank, but I know the law by heart. So does anyone who's sat through a sprint planning meeting.
At school we had it figured out, even if nobody gave it a name. Studying for the test and learning look pretty similar, right up until the day the test starts to matter more than what it measures. That's when you get the summary of the summary, the "is this going to be on the test?" question and a cheat sheet written in microscopic handwriting. The grade goes up. What you actually know, who knows.
And you can't really blame the kid, because the grade was the only thing anyone was looking at.
Teams work the same way. You measure test coverage and you get tests that don't test anything. You measure first response time in support and you get fast answers that don't solve anything. You measure number of deploys and the team starts shipping one-line changes. You measure NPS and someone asks the happy customers to please fill out the survey (not the others).
The textbook example is Wells Fargo. For years the bank pushed the goal of getting every customer to hold more and more products, and it ended up with employees opening millions of accounts nobody had asked for. The indicator looked great. The bank, not so much.
So is the way out to stop measuring? Not at all. Measuring is still the best way we have to tell whether something works, and asking what metric a feature moves still seems to me one of the most useful things you can say in a planning meeting. The trouble starts when the number stops being a signal and becomes the goal, especially if someone wins or loses something depending on how it comes out.
I don't have a recipe, but some things help. Looking at metrics in pairs, for example: velocity next to production bugs, response time next to reopened tickets. If we're closing more tickets and more bugs are showing up too, it's worth checking what we're calling done. Checking every now and then whether they still measure what we want to achieve. And not tying a bonus to a single indicator, because if your bonus depends on a number, you'll find a way to move it, even if that moves nothing else.
The most uncomfortable one of all is asking the team how they would game the metric. If the answer comes back in two seconds, it's already broken.
Goodhart is a close relative of quantophrenia, that obsession with measuring everything. Quantophrenia makes you believe that only what has a number exists, and Goodhart warns you that as soon as that number matters too much, it starts lying.
At school, a lot of the time we just wanted to pass. If on the team a green dashboard is enough, we're doing something pretty similar.