AI-built software. Expert-reviewed.

Your AI built it.Get expert eyes on it before you ship.

AI checks + human QA judgment. Get verified issues, reproduction evidence, and fix-ready prompts for your coding agent.

Instant on. Start getting AI QA value within days of our first conversation.

Human QA for AI-built software.

Start with one artifact or a focused user journey. We prioritize the most useful checks for that scope—not an exhaustive audit of your entire product.

  • Live URL or build
  • Screenshot or image
  • Document or product spec
  • API spec or collection
AI code safety

State, errors, and inputs.

False success, missing loading and recovery states, stale data, duplicate actions, partial failures, weak form validation, awkward real-world inputs, and assumptions that only work on the happy path.

Privacy and reliability

Risks behind the interface.

PII in screens, URLs, logs, or errors; unsafe permissions and defaults; exposed configuration; failed or excessive network calls; slow paths; retries; and behavior that looks complete before it actually is.

Human reality

Experience customers notice.

Fresh-eyes journey friction, real-world data and transactions, accessibility, device and channel handoffs, visual polish, brand consistency, and expert triage that makes feedback useful to developers.

QA expertise for your team

For teams

Enterprise

Private AI QA for your team, on your terms.

  • Private environment. Use your own AI keys.
  • QA experts in the loop—from IcebergQA, your own team, or both.
  • Team visibility into reviews, findings, and fixes.
IcebergQA reviewLoop

Testing experts, inside your coding agent.

Ask for testing help from your coding agent with reviewLoop. You approve what to share. AI prepares candidate findings; experts investigate and add their own testing. Feedback returns in context, ready for developer-approved fixes.

See the expert review loop How reviewLoop works
No separate briefing. No contextless bug dump. Scoped by your team · AI-prepared · Expert-tested · Returned where developers can act
AI speed. Human judgment.

Need more than one review?

Accelerate QA now.

AI-first managed testing expands your coverage with experts who run the work. Enhance your existing tests and automation—without starting over.

Learn to run it yourself.

An AI QA Capacity Sprint gives your team practical tools and habits they keep using. Continue with our experts, run it yourself, or combine both.

Latest AI QA technology from testers.ai + Experienced QA experts from XBOSoft

Before you request a review

How quickly will I get results?

We confirm availability and a turnaround with you by email before work begins. This page does not promise an instant result or a fixed delivery time.

What do I receive?

A readable review with verified issues, reproduction evidence where the artifact permits it, individual fix prompts, and a combined prompt for your coding agent. We distinguish findings from uncertainties and list areas not assessed.

Do I need to install a plugin or book a call?

Start with the short request form. You can share a URL or describe your artifact; we’ll work out access and scope by email. The coding-agent plugin is another way to bring reviewers into your workflow.

What if you don’t find bugs?

You still receive the review’s observations, checked areas, and remaining gaps. We don’t promise a minimum bug count or certify that a release is bug-free.

Can I share private software?

Describe it first. Don’t put passwords, API keys, customer data, or signed access links in this form. We’ll agree on appropriate access. Enterprise can use a private environment, your own AI keys, and your own reviewers.

The IcebergQA blog

A human perspective on AI quality.

Ideas from the people testing, steering, and taking responsibility for AI-built software.

Philip Lew · Release confidence & metrics

Confidence Is a Forecast

What does release confidence predict—and do we ever check whether it was right? Philip Lew argues for measuring the judgment behind the ship call.

Read article Read here
Phil Lew · AI tools & accountability

From Hammer to Cabinet

AI tools are getting cheaper. Phil Lew asks why clients still pay for finished work—and who takes responsibility when that work is wrong.

Read article Read here
Phil Lew · AI & human judgment

Who's Driving?

When AI writes the code, who takes the wheel? Phil Lew explores where automation helps—and where expert judgment still matters.

Read article Read here

Before you ship, get a second set of eyes.

Bring AI testing and expert judgment to your next release. Let’s talk about your team.

Human QA experts in the loop
IcebergQA / Perspectives
Release confidence & metrics

Confidence Is a Forecast

By Philip Lew · · 7 min read
A cyclist in a yellow rain jacket and helmet outdoors.

I used to live in the Netherlands, where it rains all the time and can rain any time. A friend of mine had a standard excuse for skipping a bike ride: "It could rain." My answer was always, “If we don't go riding when it could rain, we'll never go riding.”

My friend was right, of course. It could always rain. That's what made it a useless thing to say. The weather forecast was more useful. When it said forty percent chance of rain, some days I got wet and some days I didn't. So what. Forty percent meant forty percent. Unless it was actually raining, I rode anyway. I hate starting in the rain.

Software vendors don't talk like that. Go to any testing website right now, AI or not, and count how many times you see the word confidence. Ship with confidence. Release with confidence. I'm sure I've written something like that myself.

One of our clients once wanted everything automated. All 2,000 test cases. And the only thing he cared about was how many passed. Percent automated, percent passed. The tests we couldn't automate, the ones that needed a person to look at a screen and say that's wrong, didn't count for him. They weren't in the percentage, so as far as he was concerned they didn't exist.

I thought he was a weird guy. Looking back, he was just ahead of his time. He wanted a number that made him feel good about a release, and he found one.

A lot of the confidence being sold today is the same thing with AI added. I've built quality metrics for clients for a long time, so when I hear the word confidence, I want to ask two questions. Confidence in what? And how do you measure it?

Confidence is a forecast

Philip Tetlock, a psychologist then at Berkeley, spent about twenty years collecting predictions from 284 experts who made their living talking about politics and economics. By the time he published Expert Political Judgment in 2005, he had checked 82,361 of their forecasts against what actually happened. On average, the experts did only a little better than random guessing. And the experts who were on TV the most, the ones most sure of themselves, did worse than the ones nobody had heard of.

Later, in Superforecasting, which he wrote with Dan Gardner, Tetlock showed that some people really are better at predicting than others. The only way to find out who is to keep track of how their predictions turn out.

The test is simple. If you say you're 70% sure about a hundred different things, about seventy of them should happen. Weather forecasters get checked this way all the time. That's why forty percent chance of rain is worth listening to.

Now think about a release meeting. Someone asks if we're good to go. Someone says yes.

That yes is a prediction. It says this build will hold up when real customers use it. How many teams go back a month later and check whether that yes was right? How many could tell you if their yeses are right nine times out of ten, or six?

When someone sells you confidence, that's what they're selling, and yet, most never track it. The things we usually report, tests written, coverage, pass rate, defects found, all come before the decision. None of them tell you whether the decision was a good one.

When tests get cheap

There's an old rule in measurement called Goodhart's law. The anthropologist Marilyn Strathern put it best: when a measure becomes a target, it stops being a good measure. Testing has seen this for decades. Pay a team on coverage, and coverage goes up. Whether the product got any better is another question.

AI speeds that up. When a tool can write five hundred tests before lunch, test counts and coverage go up like printing money. There's more of it every week, and each one is worth less. A test suite can double in size and not protect one more user or do anything meaningful.

Last year I was teaching a metrics course with a focus on AI, and a guy asked me something that still makes me laugh. Writing test cases used to take his team eight hours. With AI it takes one. Should he tell his boss?

It's funny, but I understood why he asked. If your boss judges you by the hours, and the hours just dropped from eight to one, what happens to you? What neither of us asked was whether the one-hour test cases were any good.

He's not the only one. Google's DORA 2025 research found that 90% of software professionals now use AI at work, and more than 80% say it made them more productive. But about 30% trust what it produces only a little or not at all. So people are getting faster. Whether the work is better is harder to say, because speed is easy to count and quality isn't.

Some people suggest tracking how many AI-written tests a person has reviewed. That has two problems. First, it tells you someone looked at the test. It doesn't tell you if looking made any difference. You can get that number to 100% with people clicking approve on a Friday afternoon, and still ship the bug. The second problem? How can a tester possibly review all those tests in the first place? Nobody has time to read five hundred tests.

So what would actually tell you something?

Keeping score

If confidence is a prediction, you measure it the way Tetlock measured his experts. Write down the call, then check it later. Here's where I'd start:

  • Score the ship call. At every release, someone writes down the decision and how sure they are. A month or two later, look at what happened: bugs customers found, rollbacks, hotfixes, angry support tickets. After a few months you'll have something most teams don't have, a record of how good your judgment has been.
Illustrative forecast calibration chart: predictions of 90 percent confidence hold up about six times in ten, compared with a calibrated forecast.
Illustrative example from the article, not measured customer outcomes.
  • Count what got past you. Bugs found tell you people are busy. Bugs missed tell you where you're weak. At IcebergQA, Jason Arbon and I have been sorting test findings into three piles: what AI finds on its own, what it finds with a person's help, and what only an experienced tester finds (see Who’s Driving?). Right now it works out to roughly 79%, 13% and 8%, though that changes from project to project. My bet is that most of the bugs that reach customers come from that last 8%.
  • Track when people get pulled in. Every time a person is asked to review AI work, ask afterward: did it change anything? If the answer is always no, people stop paying attention and start clicking approve. If it catches something in that 8%, that's why you have a person there in the first place.

None of this is hard to collect. It's just uncomfortable, because it puts someone's name next to a prediction and then checks whether they were right. That's a big reason most companies would rather count tests.

It also makes the accountability conversation practical. I've written about that in Who's Driving? and From Hammer to Cabinet. Everyone agrees somebody should own the ship call. But how would you know if they're any good at it?

Back to the forecast

A forty percent chance of rain means something because weather people have been checking their forecasts against real rain for a very long time. "It could rain" never meant anything, because you can't ever prove it wrong.

A lot of the confidence being sold in software is closer to "it could rain." My client with the 2,000 test cases had a number. He never had a record of whether his releases actually held up.

So I'd ask any team, including ours: the last time you said you were confident about a release, how did it turn out? And if nobody knows, what exactly were you confident in?

Open article page
IcebergQA / Perspectives
AI tools & accountability

From Hammer to Cabinet

By Phil Lew · 6 min read
A hand holding a hammer above tools on a wooden workbench.

Twenty dollars a month

I was scrolling X on my phone when a thread from Dan Rosenthal made me stop and think. Sequoia, he wrote, thinks the next trillion-dollar company won't be a software company. Services will be the new software.

My first thought wasn't about Sequoia. It was about my Claude subscription. Twenty dollars a month.

Twenty dollars, for a tool that reads code, writes test cases, drafts a bug report, and argues with me about whether a failure is real. Not that long ago, one seat of a commercial test automation tool cost thousands of dollars a year! Now, some of the most capable tools I currently use cost less than lunch for two.

So if anyone can buy these tools for twenty dollars, why would they pay someone else to do the work?

Six dollars for every one

The thesis comes from Julien Bek at Sequoia Capital, and the math behind it is simple. For every dollar companies spend on software, they spend about six on services. Rosenthal's example makes it easy to understand. A company might pay $10,000 a year for QuickBooks and $120,000 for the accountant who closes the books. SaaS companies have spent twenty-five years fighting over the smaller number.

AI changes what you can sell. Instead of selling the tool, you sell the finished work. Traditional SaaS sells access, the client's own team runs it, and growth is capped by product and distribution. On the other hand, traditional services sell people's time and expertise by the hour, and growth is capped by hiring. The new model sells a completed piece of work. Software does most of the delivery, and the provider's people handle the decisions and the exceptions. Growth is capped by what the software can do reliably.

I've run a software testing services company since 2006. I know the hiring and labor constraints personally. Every new client meant another recruiting push, another onboarding, another team lead stretched a little thinner. Finding good testers to ramp up a new client ain’t easy.

Think about a carpenter's tools. A hammer, a saw, a chisel, a box of nails. Any hardware store sells them for a few hundred dollars, and anyone can swing a hammer. But getting from a hammer to a cabinet is a long road. You have to know which tool to pick up next, how they work together, and what to do when the wood splits. Most people who own a hammer couldn't build a cabinet with doors that close.

And nobody wants the hammer in the first place. They want the cabinet. When a cabinet door comes off its hinge, nobody calls the hardware store. They call the carpenter.

The $20 plan is the hammer.

The word that matters most

Read the last line of the new model again. Growth is capped by what the software can do reliably.

Reliably. That's a testing word. It's the whole business hiding inside one adverb.

My IcebergQA co-founder Jason Arbon makes the same point from the other direction in his book, Testing AI: Engineering Confidence in Non-Deterministic Systems. Generation got cheap, so validation has become the scarce resource. AI can produce code, reports, and decisions faster than anyone can responsibly check them. And the output that looks right but is wrong is the dangerous kind, because it passes a casual read.

So what happens to a company selling finished work when nobody can prove the work is right? It stops being a service. It becomes a liability with a monthly invoice.

Go back to the accountant. A client doesn't pay $120,000 to add up numbers. QuickBooks does the arithmetic. They pay because someone puts their name on the numbers and picks up the phone when the auditor calls. The software was never accountable. The accountant was. Take the accountant out, and what's left is fast arithmetic that nobody stands behind.

Jason has a name for the person who carries that weight: the Confidence Engineer. In his framing, the product of that job isn't a dashboard full of activity. It's a decision someone can defend later, to a client, a regulator, or a court. AI can scale the measurement work. Accountability doesn't scale the same way.

The long road to the cabinet

Cheap tools cut both ways. If the hammer costs twenty dollars, anyone can buy one. Plenty of teams already have Claude developing their code. Owning the tools isn’t the hard part.

The hard part is the road between the tool and the finished thing. Knowing how to combine a handful of tools into something that holds together, and knowing where each one falls short. Most teams don't have that skill yet, with AI or without it. That road is what clients are really paying for.

A while back I ran a small experiment. A few testers and I bug-hunted the same app that Claude was testing. On anything the spec spelled out, Claude kept up with me. On the problems nobody wrote down, it missed. It tested what it was told. It didn't stop to ask what the person who wrote the spec actually meant, or click through the app the way a confused user would.

That's where people still belong. Not running every test by hand. Deciding what's worth testing, catching what the documentation never captured, and putting their name on the call that a release is ready.

When a bug escapes to production, nobody asks which tool missed it. They ask who signed off. And “the AI did it” won’t work as an answer for long. That question doesn't go away because the testing got automated. It gets sharper, because now there's more output, produced faster, by something that can't be held responsible for any of it.

At IcebergQA, testers.ai does most of the delivery. Our people own the decisions, the exceptions, and the sign-off. The client gets finished work with someone standing behind it.

It turns out we've been building a service-as-software company without calling it that.

Who signs for the work?

If Sequoia is right, a lot of companies are about to sell finished work instead of tools. Legal work. Accounting. Support. Code. They'll all hit the same ceiling, and it won't be the price of the model or the size of the sales team. It will be whether anyone is willing to stand behind the output.

Software has always shipped with a license that says, in capital letters, we're not responsible. Services never got that luxury. If services really are the new software, that disclaimer can't come along for the ride. Somebody has to own the result.

The tools will keep getting cheaper. Twenty dollars will become ten, then something bundled into a plan nobody remembers buying. Accountability won't get cheaper. If anything, it gets more expensive every time the volume of AI output doubles.

So when your AI ships something wrong, whose name is on it?

Open article page
IcebergQA / Perspectives
AI & human judgment

Who's Driving?

By Phil Lew · 5 min read

I rode in a Tesla the other day while the driver showed me what its self-driving could do. My first thought was that I could take a nap on the way home. My second thought came a few seconds later. If it hit someone, whose fault would that be?

Ask anyone what makes them nervous about self-driving cars and you'll hear the same question within a minute.

Most people take for granted that the car can stay in its lane on a clear day. What worries them is the half second when it can't, and who answers for it afterward. The person in the driver's seat who wasn't driving? The company that wrote the software? The engineer who approved the release?

Carmakers actually grade this on a scale from 0 to 5. At 0 you do all the driving. At 5 the car doesn't need a steering wheel. The middle is where it gets interesting. At Level 2 the car steers and brakes, but you're supposed to watch every second. At Level 3 the car drives and calls you back when it gets confused. The handoff is the dangerous part. Someone who has been watching the scenery for twenty minutes doesn't become an alert driver the moment a chime goes off.

Lisanne Bainbridge saw this coming in 1983. In a short paper called "Ironies of Automation," she pointed out that the more you automate, the more you depend on the human who's left. And the less practice that human gets at the very thing you'll suddenly need them to do. Her examples came from industrial control rooms. Pull request queues didn't exist yet, but the pattern fits.

Look at where a lot of engineering teams are right now. Code arrives faster than anyone planned for. Claude, Codex and Cursor write it, often test it, and then a person is asked to approve it. That person is usually one of your best engineers, and they're looking at their fortieth PR of the week. They know they should read it carefully. They also know there are thirty-nine more to review.

So what happens? I think we all know. The approval becomes a formality. The reviewer turns into the person in the driver's seat, watching the scenery. The industry calls this Human in the Loop. Most days it's closer to Human on the Side.

And when it breaks, whose fault is it? The AI didn't sign anything. The engineer approved it in four minutes. The CTO is the one explaining the outage. We put out a guide on this called You Didn't Write That Code. You Still Own It. The title says it.

So where should the human actually be? What does Human in the Loop mean?

Jason Arbon and I have been trying to answer that for our own work first. We ended up with a grid. Down the side is the kind of quality work: quick checks, high-volume work, judgments, and investigation. Across the top is who leads, moving from AI to the expert:

  • Autonomous. A link that goes nowhere. A required field that accepts an empty value. An API response that breaks its schema. The rules are written down, so let AI run these checks and stop spending expert time on them.
  • Light guidance. Someone sets the expectations and AI does the work. Regression suites with a defined setup. Deciding which of fifty AI-flagged issues actually matter.
  • Deep guidance. AI still does the work, but an expert steers it toward bugs it would never find alone, or tunes a test agent to a codebase over weeks. This is the column teams tend to skip.
  • Expert-driven. Deciding what correct even means. The unwritten business exception everyone on the team knows and nobody documented. The fifteen seconds when a tester looks at a screen and says this feels wrong. Release sign-off.
AI and human QA collaboration matrix: quality work progresses from quick checks through high-volume work, judgments, and investigation; leadership progresses from autonomous AI through light and deep expert guidance to expert-driven work.
From passenger to driver: four levels of human involvement in quality work.

The graphic shows one square from each column. Follow them from the top left corner to the bottom right, and you can watch the person go from passenger to driver.

When we tried to put rough numbers on it, our working split came out to about 79 percent AI on its own, 13 percent shared, and 8 percent that needs an expert. Jason was careful to call that a working number, and it will move as the tools get better. But look at what it says. Most of the testing your people do by hand today shouldn't be done by hand. A small slice can't be done any other way. And that small slice is the part you'll answer for when something breaks.

In driving terms, most teams I see run in one of two modes. Either a human reviews everything, which doesn't survive the volume of AI-written code, or AI handles it and someone glances at the result, which is the car calling for help while the driver dozes. The useful work happens in the two middle columns, and it needs someone who knows the product and the tools well enough to steer.

Bainbridge's irony shows up here too. If your testers spend their days in the first column, clicking through checks a machine could run, will they have the eye ready when the fourth column needs it?

That's the work we do at IcebergQA. We add AI testing capacity with QA experts who know how to run it, steer it, and decide when a person needs to step in. You can turn it on for a release and turn it off afterward.

When your AI flags something in tomorrow’s release, who looks at it? Someone in the loop who knows within seconds whether it matters, or someone on the side, clicking approve on PR number forty?

Open article page