Confidence Is a Forecast

I used to live in the Netherlands, where it rains all the time and can rain any time. A friend of mine had a standard excuse for skipping a bike ride: "It could rain." My answer was always, “If we don't go riding when it could rain, we'll never go riding.”

My friend was right, of course. It could always rain. That's what made it a useless thing to say. The weather forecast was more useful. When it said forty percent chance of rain, some days I got wet and some days I didn't. So what. Forty percent meant forty percent. Unless it was actually raining, I rode anyway. I hate starting in the rain.

Software vendors don't talk like that. Go to any testing website right now, AI or not, and count how many times you see the word confidence. Ship with confidence. Release with confidence. I'm sure I've written something like that myself.

One of our clients once wanted everything automated. All 2,000 test cases. And the only thing he cared about was how many passed. Percent automated, percent passed. The tests we couldn't automate, the ones that needed a person to look at a screen and say that's wrong, didn't count for him. They weren't in the percentage, so as far as he was concerned they didn't exist.

I thought he was a weird guy. Looking back, he was just ahead of his time. He wanted a number that made him feel good about a release, and he found one.

A lot of the confidence being sold today is the same thing with AI added. I've built quality metrics for clients for a long time, so when I hear the word confidence, I want to ask two questions. Confidence in what? And how do you measure it?

Confidence is a forecast

Philip Tetlock, a psychologist then at Berkeley, spent about twenty years collecting predictions from 284 experts who made their living talking about politics and economics. By the time he published Expert Political Judgment in 2005, he had checked 82,361 of their forecasts against what actually happened. On average, the experts did only a little better than random guessing. And the experts who were on TV the most, the ones most sure of themselves, did worse than the ones nobody had heard of.

Later, in Superforecasting, which he wrote with Dan Gardner, Tetlock showed that some people really are better at predicting than others. The only way to find out who is to keep track of how their predictions turn out.

The test is simple. If you say you're 70% sure about a hundred different things, about seventy of them should happen. Weather forecasters get checked this way all the time. That's why forty percent chance of rain is worth listening to.

Now think about a release meeting. Someone asks if we're good to go. Someone says yes.

That yes is a prediction. It says this build will hold up when real customers use it. How many teams go back a month later and check whether that yes was right? How many could tell you if their yeses are right nine times out of ten, or six?

When someone sells you confidence, that's what they're selling, and yet, most never track it. The things we usually report, tests written, coverage, pass rate, defects found, all come before the decision. None of them tell you whether the decision was a good one.

When tests get cheap

There's an old rule in measurement called Goodhart's law. The anthropologist Marilyn Strathern put it best: when a measure becomes a target, it stops being a good measure. Testing has seen this for decades. Pay a team on coverage, and coverage goes up. Whether the product got any better is another question.

AI speeds that up. When a tool can write five hundred tests before lunch, test counts and coverage go up like printing money. There's more of it every week, and each one is worth less. A test suite can double in size and not protect one more user or do anything meaningful.

Last year I was teaching a metrics course with a focus on AI, and a guy asked me something that still makes me laugh. Writing test cases used to take his team eight hours. With AI it takes one. Should he tell his boss?

It's funny, but I understood why he asked. If your boss judges you by the hours, and the hours just dropped from eight to one, what happens to you? What neither of us asked was whether the one-hour test cases were any good.

He's not the only one. Google's DORA 2025 research found that 90% of software professionals now use AI at work, and more than 80% say it made them more productive. But about 30% trust what it produces only a little or not at all. So people are getting faster. Whether the work is better is harder to say, because speed is easy to count and quality isn't.

Some people suggest tracking how many AI-written tests a person has reviewed. That has two problems. First, it tells you someone looked at the test. It doesn't tell you if looking made any difference. You can get that number to 100% with people clicking approve on a Friday afternoon, and still ship the bug. The second problem? How can a tester possibly review all those tests in the first place? Nobody has time to read five hundred tests.

So what would actually tell you something?

Keeping score

If confidence is a prediction, you measure it the way Tetlock measured his experts. Write down the call, then check it later. Here's where I'd start:

• Score the ship call. At every release, someone writes down the decision and how sure they are. A month or two later, look at what happened: bugs customers found, rollbacks, hotfixes, angry support tickets. After a few months you'll have something most teams don't have, a record of how good your judgment has been.

• Count what got past you. Bugs found tell you people are busy. Bugs missed tell you where you're weak. At IcebergQA, Jason Arbon and I have been sorting test findings into three piles: what AI finds on its own, what it finds with a person's help, and what only an experienced tester finds (see Who’s Driving?). Right now it works out to roughly 79%, 13% and 8%, though that changes from project to project. My bet is that most of the bugs that reach customers come from that last 8%.
• Track when people get pulled in. Every time a person is asked to review AI work, ask afterward: did it change anything? If the answer is always no, people stop paying attention and start clicking approve. If it catches something in that 8%, that's why you have a person there in the first place.

None of this is hard to collect. It's just uncomfortable, because it puts someone's name next to a prediction and then checks whether they were right. That's a big reason most companies would rather count tests.

It also makes the accountability conversation practical. I've written about that in Who's Driving? and From Hammer to Cabinet. Everyone agrees somebody should own the ship call. But how would you know if they're any good at it?

Back to the forecast

A forty percent chance of rain means something because weather people have been checking their forecasts against real rain for a very long time. "It could rain" never meant anything, because you can't ever prove it wrong.

A lot of the confidence being sold in software is closer to "it could rain." My client with the 2,000 test cases had a number. He never had a record of whether his releases actually held up.

So I'd ask any team, including ours: the last time you said you were confident about a release, how did it turn out? And if nobody knows, what exactly were you confident in?