Overview With Examples

Think of this series as a shift from checking one answer to measuring a behavior pattern. A chatbot, recommendation engine, summarizer, fraud model, or agent can look good in one demo and still fail too often across real traffic. The work is to measure that behavior across enough examples to make a responsible decision.

For example, one refund answer may be perfect, but the next hundred answers may reveal policy confusion, uneven tone, and a few dangerous promises. The next-generation AI builder sees the distribution, not just the demo.

Testing used to be simpler. You gave software an input, checked the output, and decided whether the result matched your expectation. That model still matters. A login form should still reject a bad password. A calculator should still return the same sum. A checkout flow should still charge the correct amount.

But that is no longer the whole testing world.

Modern products increasingly include systems that do not behave the same way twice. LLMs may answer the same question twice, in different words, with the same meaning and impact. Recommendation engines may change ranking order. ML models may drift after retraining. AI agents may use different paths and tools to complete the same task. Distributed services may process events in different orders depending on timing.

For developers building AI features, this changes the center of gravity. The question is no longer only, "Did the system return the expected answer?" The better question is, "Across a realistic sample of cases, how often does this system behave acceptably, how bad are the failures, and how confident are we in that estimate?"

That sounds mathematical, but it does not require becoming a statistician. It requires a practical testing mindset. You need to test at scale because individual examples can mislead you. You need to understand sampling because one run tells you almost nothing. You need to understand variance because not every difference is a bug. You need to understand confidence intervals because sample results are estimates, not exact truth. You need to understand p-values and t-tests well enough to compare versions without fooling yourself.

LLMs can help with this work. They can judge outputs against rubrics, summarize failures, cluster similar issues, compare two responses, and even help explain statistical results. But the builder still owns the judgment. The builder defines the rubric. The builder chooses the sample. The builder watches for rare failures. The builder decides whether the evidence is strong enough to ship. And yes, more often the builder is also the AI, which also needs these skills.

A next-generation AI builder does not say, "I tried it once and it worked." They can say something more useful: "We tested 300 realistic cases. The average quality score improved from 7.6 to 8.2. The 95% confidence interval for the improvement is +0.3 to +0.9. Policy failure rate dropped from 6% to 2%. No critical safety failures were observed. Recommendation: ship with post-release monitoring."

That is a different level of quality conversation. It gives product leaders, engineers, and compliance teams evidence they can reason about. It also gives developers a more strategic way to own AI behavior instead of tossing uncertainty over the wall.

This series is about that shift. Each article introduces one concept developers, testers, and AI builders can use to evaluate non-deterministic systems: rubrics, scoring, sampling, confidence intervals, t-tests, p-values, metamorphic testing, stratified reporting, rare failure hunting, release gates, and monitoring.

The goal is not to make testing colder or more mechanical. It is to make judgment clearer. When systems are unpredictable, quality does not come from pretending uncertainty is gone. Quality comes from measuring uncertainty honestly and deciding what level of risk is acceptable.

Examples

To make the book easier to use, the chapters keep returning to three familiar product examples: web search, chatbots, and AI coding agents.

Web Search Example

Web search is a useful example because quality is not one answer. A search system has to understand intent, rank relevant results, avoid unsafe or spammy pages, handle freshness, diversify results, respect latency, and improve without making long-tail queries worse.

In a real test plan, that means creating query slices with expected relevant documents, unacceptable results, freshness expectations, safety rules, and ranking metrics. The builder is not asking whether one result page looked good. The builder is asking whether the whole result set still serves user intent across common, ambiguous, long-tail, adversarial, and high-value queries.

Chatbot Example

Chatbots are useful for the same reason in a different shape. A chatbot has to answer correctly, stay grounded, use the right tone, remember the right context, refuse unsafe requests, recover from confusion, and sometimes call tools or escalate to a person.

In practice, evaluate complete conversations, not isolated messages. A useful chatbot eval captures the user's goal, the required policy or source facts, the acceptable range of answers, refusal or escalation rules, tone expectations, and whether the conversation actually resolves the user's problem.

AI Coding Agent Example

AI coding agents are the third example because they make validation painfully concrete. A coding agent can generate files, edit tests, call tools, refactor architecture, and appear productive while quietly introducing bugs, security holes, brittle abstractions, or false confidence.

For coding agents, the example case should include the task, the repo state, the files the agent should inspect, the tests it should run, the changes it should avoid, and the review rubric for correctness, security, maintainability, and blast radius. The output is not just code. It is a trace of decisions that must earn trust.

These three examples give readers something concrete to hold onto. When a chapter explains sampling, confidence intervals, LLM judges, evals, RAG, bias, rollouts, cost, or safety, the examples show how the idea applies to a ranked-results product, a conversational product, and a code-generating agent.

Testing/Quality Example

A useful quality example is a support assistant evaluated on 300 recent customer questions. Each answer gets a 0-10 quality score, a policy-pass flag, a safety flag, and a short failure category. The report does not say, "it worked." It says the average score, the confidence interval, the failure rate, the worst observed output, and the ship recommendation.

Expert Notes

At an expert level, the main move is separating observation from inference. The sample result is what you saw. The confidence interval is what you estimate about the wider population. The release decision is a risk judgment that uses both, plus business context, severity, reversibility, and monitoring plans.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Non-determinism means repeated runs can produce different behavior, even when the input looks the same. That can happen because of model sampling, personalization, ranking experiments, timing, cache state, tool calls, retrieved data, or hidden production context.

For example, an LLM may choose different words, a search system may reorder equivalent results, and a distributed service may process two events in different orders. Some of that variation is harmless. Some of it changes the truth.

It is important not to treat "non-deterministic" as one simple category. Some stochastic systems can be made mostly reproducible by fixing the random seed, data snapshot, configuration, and runtime. That is useful for debugging and for reducing sample counts when you are validating a narrow behavior. LLM-based products are trickier. Even with low temperature, they may vary because the provider changed the served model, a safety layer changed, a tool returned different data, retrieval context shifted, hidden state changed, floating-point or hardware behavior differed, or the platform routed the request through a different path. The testing strategy depends on which kind of variation you are trying to control.

A deterministic system gives the same output every time you provide the same input under the same conditions. A calculator is the easiest example. If you enter 2 + 2, you expect 4 every time. If a deterministic API receives the same request with the same database state and configuration, you expect the same response.

A non-deterministic system is different. The same input may produce different outputs. Sometimes that variation is intentional. Sometimes it is a side effect of timing, randomness, personalization, or hidden state. Sometimes it is a bug.

LLMs are the most visible example. Ask the same model to summarize a document ten times and you may get ten different summaries. Some differences are harmless. The model may choose different wording, sentence order, or examples. Other differences are serious. One summary may omit a key risk, invent a fact, or contradict the source material.

Recommendation systems are also non-deterministic from the tester's point of view. The same user might see different products depending on inventory, ranking experiments, recency, or personalization signals. Search systems may reorder results as indexes update. Fraud models may return slightly different risk scores after retraining. AI agents may call tools in different sequences while still completing the same task.

Distributed systems add another flavor of non-determinism. Events may arrive in different orders. A cache may be warm or cold. A retry may succeed or fail depending on timing. Two services may race. The code may be deterministic locally, but the system behavior is not perfectly repeatable in production.

This matters because traditional testing often assumes a single expected output. That is still appropriate for many parts of a product, or component or unit test, but it is not enough for systems with acceptable variation. For those systems, testers need to define what must remain stable even when surface behavior changes.

For example, an LLM support assistant may phrase a refund answer in different ways. That is acceptable if the policy stays correct. It is not acceptable if one response says returns are allowed within 30 days and another says 45 days. The words can vary. The business rule cannot.

The tester's job is to separate variation from failure. Wording variance may be healthy. Formatting variance may be tolerable. Factual variance, safety variance, privacy variance, and policy variance may be release blockers.

That is the first mental shift in testing non-deterministic systems. You are not only checking one answer. You are evaluating a range of possible behaviors and deciding whether that range is safe, useful, and trustworthy enough for users.

Examples

Web Search Example

A web search engine may return slightly different rankings as indexes refresh, personalization changes, ads rotate, or ranking features update. The test is whether the most useful content for the intent still rises to the top, not whether every result appears in the same slot forever.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A chatbot may answer the same question twice, in different words, with the same meaning and impact. The test is whether the answer remains correct, grounded, safe, and useful, not whether the sentence is identical.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An AI coding agent may solve the same ticket with different edits, helper functions, or file boundaries. The test is whether the behavior, maintainability, and safety hold, not whether the patch looks identical.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality test for a refund assistant should not fail because one answer says "30 days" in the first sentence and another says it in the second. It should fail if the answer changes the policy, invents an exception, omits a required disclosure, or gives the user a next step that support cannot honor.

Expert Notes

Technically, testers should separate sources of randomness from sources of state and sources of platform change. Model temperature, random seeds, ranking tie-breakers, async timing, retrieval snapshots, feature flags, user profiles, tool outputs, provider model versions, safety filters, hardware/runtime paths, and hidden product context should be logged independently because each one creates a different debugging path.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Exact assertions are still valuable, but fuzzy outputs need criteria. The important question becomes: what properties must every acceptable output preserve? Those properties may include factual correctness, policy compliance, completeness, tone, safety, citation quality, or refusal behavior.

For example, two summaries can use different wording and both be good if they preserve the same facts. Two support answers can sound different and both be good if they follow the same policy.

Exact assertions are one of the great strengths of software testing. If a function should return 42, the test should assert 42. If a checkout flow should charge $19.99, the test should verify $19.99. When correctness is exact, exact tests are appropriate.

Non-deterministic outputs often need a different approach.

Imagine a support assistant answering a refund question. The expected answer might be, "No, shoes can only be returned within 30 days." But the system replies, "Returns are available for 30 days after purchase, so a 45-day return is outside the standard window." A strict string comparison would fail that answer, even though the product behavior is good.

The problem is that the test is checking the sentence rather than the property that matters. The property is policy correctness. The answer should communicate the 30-day limit, avoid inventing exceptions, and give the user a clear next step. The exact wording is secondary.

This is where evaluation criteria become essential. Instead of defining one expected output, testers define the characteristics of an acceptable output. For the refund example, the criteria might say: the answer must state that returns are allowed within 30 days only; it must not imply that a 45-day return is probably accepted; it should be direct and polite; it should not promise that support can override the policy unless that is documented.

Those criteria can be checked by humans, by deterministic rules, by an LLM judge, or by a combination of methods. The important part is that the test now matches the real quality question.

Evaluation criteria also make failures easier to discuss. Instead of saying, "The answer did not match the expected string," the tester can say, "The answer failed because it suggested an unsupported policy exception." That is much more useful to the team.

This does not mean exact assertions disappear. Some requirements should remain hard checks. A system must not leak private data. It must not make up prices. It must not execute an unsafe action. It must not omit required compliance language. When the rule is absolute, the test should be absolute.

A mature non-deterministic test strategy uses both styles. Exact assertions protect hard boundaries. Evaluation criteria measure flexible quality. The art is knowing which parts of the behavior may vary and which parts must remain fixed.

That shift makes the test suite less brittle and more aligned with user trust. The goal is not to force every output into the same shape. The goal is to make sure every acceptable output preserves the facts, constraints, and safety rules that matter.

Examples

Web Search Example

A good rubric separates relevance, freshness, authority, diversity, safety, and result presentation. A result set can score high even when two acceptable pages swap positions.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A good rubric separates correctness, completeness, grounding, tone, refusal behavior, and actionability. A fluent answer should not receive a high score if it invents policy or misses the user's real need.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

A good rubric separates functional correctness, test quality, minimality, security, maintainability, integration risk, and whether the agent changed code it should have left alone.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is replacing an expected-string check with a rubric: the answer must state the 30-day limit, must not promise a manual override, must give a clear next step, and must use respectful language. The answer can vary, but the required properties cannot.

Expert Notes

Expert teams usually split criteria into hard constraints and soft quality dimensions. Hard constraints are binary blockers, such as no private data leakage. Soft dimensions can be scored, such as clarity or completeness. Mixing the two into one score hides the failures that should stop release immediately.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

A 0-10 score turns fuzzy judgment into data the team can trend, compare, and discuss. The score does not remove subjectivity; it makes subjectivity explicit enough to calibrate.

For example, a score of 9 might mean correct, complete, safe, and polished. A score of 6 might mean basically useful but incomplete. A score of 2 might mean misleading, unsafe, or unusable.

Pass/fail is useful, but it is sometimes too blunt for non-deterministic systems.

An LLM answer may be correct but vague. A recommendation list may include useful items but miss the best one. A generated summary may be accurate but too long. A search result may contain the right answer, but rank it lower than users need. Calling all of these simply "pass" or "fail" loses important information.

A 0-10 scoring scale gives testers a way to measure degrees of quality. It does not make judgment perfect, but it makes judgment visible, repeatable, and discussable.

The scale should be defined before testing begins. A score of 10 should mean the output is excellent: correct, complete, safe, clear, and ready to ship. Scores of 8 or 9 should mean the output is good, with only minor issues. Scores of 6 or 7 might be acceptable but not ideal. Scores of 4 or 5 indicate weak, incomplete, confusing, or risky behavior. Scores from 0 to 3 should represent severe failures: misleading, unsafe, unusable, or catastrophic.

The exact rubric should match the product. A support assistant should be judged on policy compliance, helpfulness, tone, and correctness. A medical summarization tool should be judged much more strictly on factual accuracy and omission risk. A creative writing assistant may tolerate more stylistic variation, but still needs safety and relevance criteria.

The power of numeric scoring appears when you sample many outputs. You can calculate an average score, but you can also look at the minimum score, the percentage of outputs above a threshold, and the rate of unacceptable failures. A system with scores of 8, 8, 8, 8 is very different from one with scores of 10, 10, 4, 8, even if the averages are similar.

Release gates can use scores in practical ways. A team might require an average score of at least 8.0, at least 95% of outputs scoring 7 or above, no output below 4, and no critical safety failure. This combines a quality target with protection against bad tails.

The worst score deserves special attention. If 99 outputs score 9 and one output scores 0 because it leaks private data, the average will still look excellent. That does not mean the system is safe. Scores help summarize quality, but hard failures must still block release.

Scoring also helps compare versions. If a new prompt raises the average score from 7.6 to 8.2 and reduces low-scoring outputs, that is meaningful evidence. If the average rises but the worst cases get worse, the team should slow down.

A 0-10 score is not magic. It is a practical measurement language. It gives testers, engineers, and product leaders a shared way to talk about fuzzy output quality without pretending it is purely binary.

Examples

Web Search Example

A good rubric separates relevance, freshness, authority, diversity, safety, and result presentation. A result set can score high even when two acceptable pages swap positions.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A good rubric separates correctness, completeness, grounding, tone, refusal behavior, and actionability. A fluent answer should not receive a high score if it invents policy or misses the user's real need.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

A good rubric separates functional correctness, test quality, minimality, security, maintainability, integration risk, and whether the agent changed code it should have left alone.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is scoring 200 generated answers with a rubric that separates hard failures from quality levels. The release report can show average score, median score, percentage scoring 7 or higher, number below 4, and the worst observed output with its failure reason.

Expert Notes

At expert level, define anchor examples before scoring begins. Reviewers need concrete examples of a 10, 7, 4, and 0. Without anchors, scores drift over time and different reviewers quietly apply different scales.

Older machine-learning evaluation systems often express quality as values between 0 and 1: 0.54, 0.71, 0.93, and so on. That can be useful when training a neural network, optimizing a loss function, or feeding a metric into another mathematical system. But for LLM output review, human judgment, rubric scoring, and product release decisions, that apparent precision is often fake. People and language do not naturally operate at the difference between 0.54 and 0.55, and LLM judges do not reliably mean something stable at that tiny decimal step either.

That is why practical AI quality work is moving toward scales like 0-10, anchored by examples. A 7 can mean "useful but incomplete." A 4 can mean "weak or risky." A 0 can mean "severe failure." Those categories are easier for people and LLM judges to apply consistently. Unless you are training a model or optimizing a numeric loss directly, the extra decimal precision usually does not add meaning. It often just makes a fuzzy judgment look more scientific than it is.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

An LLM judge is an evaluator model. It reviews an input, an output, relevant context, and a rubric, then produces a score, labels, or explanation. It is useful when exact assertions cannot capture the quality question.

For example, a judge can evaluate whether a support answer follows policy, whether a summary is faithful to a source document, or whether two candidate responses differ in safety and usefulness.

An LLM judge is a model used to evaluate another system's output. The judge receives the input, the output, relevant policy or source material, and a rubric. It then scores the output and explains the reason for the score.

This pattern is especially useful when outputs are natural language. A deterministic test can easily check whether a field is present or a JSON schema is valid. It is much harder to check whether a support answer is clear, faithful to policy, complete, and appropriately cautious. An LLM judge can help with that kind of evaluation at scale.

A major reason this pattern caught on is that strong judges can agree surprisingly well with humans. The 2023 Berkeley/LMSYS paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" found that GPT-4 judge agreement with human preferences could exceed 80% in their MT-Bench and Chatbot Arena settings, roughly in the range of human-human agreement. That does not make LLM judges perfect, but it shows they can be credible measurement tools when the task, rubric, and calibration are well designed.

The other major reason is economics. LLM judges are fast, cheap, repeatable, and scalable compared with human review. That matters because cheaper evaluation means teams can run more tests, cover more slices, compare more versions, audit more production traces, and catch more regressions before users do. Even when an LLM judge is not better than a trained human for a single decision, it can make the whole quality system better by making far more measurement possible. Over time, strong judges will become better than human judges in many domains because they can be calibrated on more examples, stay consistent across large batches, and combine policy, examples, source documents, and historical failure patterns without fatigue.

A basic judge prompt might include the user's question, the system's answer, the official policy, and instructions such as: evaluate whether the answer follows the policy; score it from 0 to 10; identify any hard failures; explain the score briefly; report whether the judgment is high, medium, or low confidence.

For example, suppose the policy says returns are accepted within 30 days only. The user asks whether shoes can be returned after 45 days. The system answers, "You can probably return them if they are unused." A good judge should give that answer a low score because it contradicts the policy and invents an unsupported exception.

LLM judges can do more than assign scores. They can compare two outputs, summarize common failure patterns, cluster related issues, flag borderline examples for human review, and explain why a particular answer is risky. This makes it possible to evaluate hundreds or thousands of outputs that would be too expensive to review manually.

But an LLM judge is not an oracle. It is also a non-deterministic system. It can be inconsistent. It can be too lenient. It can be fooled by fluent but wrong answers. It can miss domain-specific rules. It can disagree with expert humans.

That is why judge design is a testing problem of its own. The rubric should be explicit. The prompt should include examples of strong, weak, and failing outputs. Hard failures should be clearly separated from quality preferences. For important domains, judge results should be calibrated against human reviewers.

A useful practice is to review disagreements between the LLM judge and human testers. If the judge consistently gives high scores to answers that humans consider risky, the rubric or judge prompt needs work. If humans disagree with each other, the policy or rubric may be unclear.

It is especially important to analyze agreement and disagreement by data slice. Overall agreement can look healthy while hiding the fact that the LLM judge is better on simple English support answers, humans are better on subtle policy violations, domain experts are better on medical or legal cases, and native speakers are better on regional language or culture. AI judges and human raters often get different things right and wrong. The useful signal is not only "how often do they agree?" It is "where do they agree, where do they disagree, who is right in each slice, and what does that say about the rubric, judge, rater pool, and release risk?"

LLM judges work best as part of a layered evaluation system. Deterministic checks catch schema problems, prohibited phrases, missing citations, or hard policy violations where possible. LLM judges evaluate fuzzy quality. Humans review samples, critical failures, and ambiguous cases.

The tester remains responsible for the evaluation design. The LLM judge can scale review, but it should not own the definition of quality or the decision to ship.

Examples

Web Search Example

An LLM judge can review a query and result list, score whether the top results satisfy intent, and explain why a result is irrelevant, stale, spammy, or unsafe.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Back in the day, the search engine I worked on was called MSN Live Search. We had a list of things we called "definitives," which sounded smart and official, but in practice meant a hard-coded list of queries where the search engine should always return a specific result. If someone searched for "Microsoft search," the first result should be MSN Live Search at the top. That sounds sensible until the product is rebranded as Live Search, and then searches for "live search" or "Microsoft search" keep pointing users to the previous version of the product, old branding, or broken links. Dynamic outputs demand dynamic and intelligent validation systems.

Chatbot Example

An LLM judge can score an answer against a rubric, compare two candidate responses, identify unsupported claims, and flag tone or policy problems for human review.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An LLM judge can review a diff, summarize risk, spot likely missing tests, compare approaches, and flag suspicious code, but it still needs executable checks and human calibration.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is running 1,000 assistant answers through a judge that returns score, policy-pass, safety-pass, confidence, and a short reason. Human reviewers then audit a sample of high-risk cases and judge-disagreement cases before trusting the aggregate report.

Expert Notes

Expert teams test the judge as a system under test. They measure agreement with human reviewers, track bias toward fluent answers, use blinded comparisons, keep judge prompts versioned, and quarantine examples where the judge is low-confidence or historically unreliable. The Berkeley/LMSYS result is encouraging, but it should be treated as evidence for careful judge design, not permission to skip calibration.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Variance is the spread of behavior. In non-deterministic systems, some spread is expected. The tester's job is deciding which spread is healthy flexibility and which spread is quality risk.

For example, wording variance may be acceptable in a support answer, but factual variance is not. Latency variance may be acceptable for a batch report but unacceptable for a real-time assistant.

Non-deterministic systems vary. That is expected. The tester's job is not to eliminate all variation. The tester's job is to understand which variation is acceptable and which variation is dangerous.

Wording variance is often harmless. If one answer says, "Your refund was approved," and another says, "We approved your refund," the user receives the same information. A strict text comparison might notice the difference, but a quality evaluation should probably treat both answers as acceptable.

Structural variance can also be acceptable. One response may use bullets, while another uses a paragraph. One summary may start with the conclusion, while another starts with background. This becomes a problem only when the product requires a specific format or when the structure makes the answer less usable.

Factual variance is much more serious. If one answer says returns are allowed within 30 days and another says 45 days, the system is not merely varying its style. It is changing the truth. For a support product, that can create customer frustration, financial loss, and loss of trust.

Safety variance can be even more important. A model that usually refuses an unsafe request but occasionally provides instructions has a tail-risk problem. The average behavior may look good, but the rare failure may be unacceptable.

Latency variance is another common form. A service may usually respond in 500 milliseconds but sometimes take 10 seconds. Users experience the tail, not the average. For real-time products, those spikes may matter as much as correctness.

Ranking variance requires its own judgment. A search system may return the same relevant results in different orders. That may be fine if users still find what they need. It may be a problem if the best result frequently falls below the visible fold or if important safety information gets buried.

Before testing, teams should define acceptable variance. Wording may vary. Formatting may vary within limits. Required facts may not vary. Prohibited content may never appear. Response time must stay within a threshold. Required items must appear in the top N results.

This helps teams avoid two bad extremes. One extreme is treating every difference as a failure, which makes the test suite brittle and noisy. The other is accepting all variation as normal, which hides real quality problems.

Non-deterministic testing is the discipline of drawing that line clearly. Variation is not the enemy. Harmful, uncontrolled variation is.

Examples

Web Search Example

A web search engine may return slightly different rankings as indexes refresh, personalization changes, ads rotate, or ranking features update. The test is whether the most useful content for the intent still rises to the top, not whether every result appears in the same slot forever.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A chatbot may answer the same question twice, in different words, with the same meaning and impact. The test is whether the answer remains correct, grounded, safe, and useful, not whether the sentence is identical.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An AI coding agent may solve the same ticket with different edits, helper functions, or file boundaries. The test is whether the behavior, maintainability, and safety hold, not whether the patch looks identical.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is repeating the same 50 high-risk prompts 10 times each. The tester tracks whether required facts stay stable, whether refusal behavior changes, whether latency has a long tail, and whether any run crosses a hard safety boundary.

Expert Notes

At expert level, variance should be reported by dimension. Score variance, factual variance, latency variance, ranking variance, and policy variance are not interchangeable. A single average can hide the type of instability users will actually feel.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

Sampling is the first bit of math in the book, but the idea is familiar: do not judge a restaurant from one bite, a movie from one scene, or an AI system from one lucky answer.

A sample is just the set of examples you looked at. The math helps you remember that the examples you saw are not the same thing as the whole future behavior of the product. Before formulas matter, the practical question is simple: did we look at enough of the right examples to make a responsible decision?

Overview With Examples

Sampling is how testers turn scattered observations into evidence. One output is an anecdote. A sample lets you estimate how the system behaves across a wider set of users, inputs, and conditions.

For example, one good answer does not prove a chatbot is good, and one bad answer does not prove it is broken everywhere. A sample shows how common each outcome is.

A single test run can be useful for debugging, but it tells you very little about the quality of a non-deterministic system.

Suppose you ask an LLM support assistant one refund question and it gives an excellent answer. That is nice, but it does not prove the assistant is reliable. The next answer may be vague. The third may be correct. The fourth may hallucinate a policy exception. If you stop after one run, you will never see the pattern.

The same is true in the other direction. One bad output does not tell you whether the system is terrible or whether you found a rare edge case. The failure matters, but you need sampling to estimate how common it is.

Sampling means running enough tests to observe a distribution of behavior. You might repeat the same prompt many times to measure stability. You might test many realistic prompts once to measure coverage. You might do both: repeat known risky cases and also sample fresh real-world inputs.

Different sampling dimensions reveal different risks. Repeating the same prompt reveals randomness for that case. Sampling across many prompts reveals coverage across user needs. Sampling across languages may reveal localization problems. Sampling across personas may reveal tone or accessibility gaps. Sampling high-risk edge cases may reveal rare but severe failures.

A useful test plan often includes several layers. First, run a small smoke sample to catch obvious problems. Then run a larger evaluation set to estimate product quality. Then run targeted stress tests for safety, privacy, and policy boundaries. Finally, continue sampling after release to detect drift.

Sampling also changes how teams talk about results. Instead of saying, "The model gave a good answer," you can say, "Across 100 refund-policy cases, the average score was 8.4, the failure rate was 3%, and the worst output incorrectly allowed a late return." That is a much stronger quality statement.

The important idea is simple: one run tells you what happened once. A sample tells you how the system behaves. Non-deterministic testing begins when testers stop treating a single output as the whole truth.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, one sample tells almost nothing because disease prevalence, scanner type, image quality, patient demographics, and rare pathology all change the meaning of the result. A demo case where the model spots one tumor is not evidence that it is safe across clinics, populations, and edge cases. Build samples that include normal images, ambiguous images, rare findings, artifacts, and known hard negatives.

Humanoid Robot Example

For humanoid robots and embodied AI, one successful demo walk, handoff, or door-opening task is not enough. Test across floor surfaces, lighting, occlusions, human movement, sensor noise, battery state, and unexpected interruptions. The sample must include near-miss conditions, not just polished demonstrations.

Testing/Quality Example

A testing/quality example is building a sample with 100 common questions, 50 edge cases, 25 adversarial prompts, and 25 recent production-like inputs. The report separates results by category instead of blending them into one vague score.

Expert Notes

Expert sampling plans specify the population, sampling frame, inclusion criteria, exclusions, randomization method, and known bias. If the sample only includes easy happy-path prompts, the confidence interval describes easy happy-path prompts, not the product.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

Sample size sounds like a formula problem, but it begins as a decision problem. The higher the stakes, the smaller the expected improvement, or the rarer the failure, the more evidence you need.

You do not need to memorize a sample-size equation to use this idea. Start by asking what mistake would be expensive: shipping a bad system, blocking a good one, missing a rare severe failure, or spending too much time measuring tiny differences that do not matter.

Overview With Examples

Enough samples means enough evidence for the decision and risk. Low-risk changes can use smaller samples. High-impact systems need larger samples, deeper review, and targeted tests for rare but severe failures.

For example, a creative rewrite tool and a billing agent should not use the same sample-size bar. The cost of being wrong is different.

The most common question in non-deterministic testing is also the hardest: how many samples are enough?

There is no universal answer. The right sample size depends on the decision you need to make, the risk of the feature, the variability of the system, and the failure rate you are trying to detect.

A small sample can be useful. Ten examples may be enough for a quick smoke check. If the system fails obviously in ten runs, you do not need a larger study to know there is a problem. Thirty examples can provide a rough directional read, especially early in development. One hundred examples can produce a more useful product-quality estimate. Hundreds of examples are more appropriate for release gates. Thousands may be necessary for rare failure hunting or production monitoring.

The key is to match sampling effort to risk. A low-risk creative writing feature may not need hundreds of examples before every change. A billing assistant, medical summary, legal advice boundary, refund policy flow, or account deletion agent deserves far more evidence.

Rare failures require special attention. If a dangerous failure happens 1% of the time, a sample of 30 may easily miss it. If a privacy leak happens 0.1% of the time, even hundreds of samples may not be enough to observe it reliably. That does not mean testing is hopeless. It means random sampling should be combined with targeted stress tests designed to provoke the risky behavior.

Zero observed failures does not mean zero risk. If you run 30 tests and see no failures, you have learned that failures did not appear in that sample. You have not proven that the true failure rate is zero. This distinction is crucial when teams are tempted to overclaim based on small samples.

Variability also affects sample size. If scores are tightly clustered, fewer samples may estimate average quality reasonably well. If scores swing from excellent to terrible, you need more samples to understand the distribution.

A practical rule is to ask three questions. First, how bad would it be if this system failed? Second, how rare a failure do we need to detect? Third, how narrow does our uncertainty need to be before making a ship decision?

Enough samples means enough evidence for the decision at hand. Testers do not need perfect certainty. They need a defensible level of confidence for the risk they are accepting.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, sample size depends on the decision. A quick smoke test can catch obvious model or pipeline failures, but release evidence needs enough positive and negative cases to estimate sensitivity and specificity. Rare conditions need enriched test sets or targeted case collection because a random sample may contain too few true positives to say anything useful.

Testing/Quality Example

A quality example is using 50 cases for a quick prompt smoke check, 200-500 cases for a release comparison, and a separate targeted set for policy, privacy, safety, or irreversible actions. The sample count is tied to the decision being made.

Expert Notes

At expert level, sample size depends on desired precision, expected failure rate, confidence level, and minimum detectable effect. Rare failures need targeted hunting because random sampling can require impractically large counts to observe very low-frequency events.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

The goal of basic statistics is not to make engineering feel academic. The goal is to give names to things builders already notice: the typical result, the worst result, the spread, the tail, and the rate of bad outcomes.

If the formulas feel intimidating, translate each number back into a product question. Mean asks, "How good is it on average?" Minimum asks, "How bad did it get?" Failure rate asks, "How often did it cross a line we care about?" Percentiles ask, "What happens to the unlucky users?"

Overview With Examples

A few basic statistics make non-deterministic quality visible. Mean, median, percentiles, standard deviation, failure rate, and minimum score each answer a different quality question.

For example, the mean tells the overall level, the median tells the typical case, percentiles show the tail, and the minimum tells whether something truly bad happened.

Developers do not need to become statisticians to evaluate non-deterministic systems. But a few basic measures can make quality reports much more useful.

The mean is the average score. If five outputs score 8, 9, 7, 8, and 8, the mean is 8.0. The mean is helpful because it summarizes overall quality, but it can hide bad outliers. A system that usually performs well but occasionally fails catastrophically may still have a high mean.

The median is the middle score. It is useful when outliers distort the average. If the scores are 2, 8, 8, 9, and 9, the median is 8. The median tells you the typical result is good, while the score of 2 tells you there is still a serious tail problem.

The minimum is the worst observed score. For safety-sensitive systems, this may be one of the most important numbers in the report. An average score of 8.7 sounds strong. A minimum score of 0 because the system leaked private data is a release blocker.

Failure rate measures how often outputs violate a threshold or hard rule. If 5 out of 100 outputs fail, the observed failure rate is 5%. For policy, privacy, safety, and compliance testing, failure rate may be more important than average quality.

Standard deviation measures how spread out the scores are. A system that scores 8, 8, 8, and 8 is stable. A system that scores 10, 10, 4, and 8 may have a similar average, but it is less predictable. High variability means users may have very different experiences.

Percentiles help builders understand tails. P95 latency means 95% of requests were faster than that value. P5 quality means 5% of outputs scored at or below that level. Percentiles are useful because users often remember the worst experiences, not the average.

A strong report uses several of these measures together. It might say: the average score was 8.4, the median was 8.6, the worst score was 3, the failure rate was 4%, and the lowest-scoring category was policy edge cases. That tells a richer story than the average alone.

The lesson is not that every developer needs complex statistics. The lesson is that one number is rarely enough. Non-deterministic quality lives in the spread, the tails, and the failure patterns.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, basic stats are not academic decoration. Sensitivity answers how often the model catches real disease. Specificity answers how often it avoids false alarms. Positive predictive value changes with prevalence, so the same model can look useful in one clinic and noisy in another. Report the confusion matrix, not only accuracy.

Testing/Quality Example

A testing/quality example is reporting: mean score 8.2, median 8.5, p5 score 4.0, minimum score 1, policy failure rate 3%, and worst category "billing edge cases." That is much more useful than saying the model averaged 8.2.

Expert Notes

Expert reports avoid letting one metric dominate. For skewed distributions, the median and percentiles may explain user experience better than the mean. For safety-sensitive systems, the tail and failure rate often matter more than average quality.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

A confidence interval is math's way of adding humility to a measurement. It keeps the team from treating one sample as if it were the whole universe.

Before worrying about how the interval is calculated, focus on what it does for the conversation. It turns "the score is 8.2" into "based on the sample, the real score is probably around here." That small shift prevents a lot of overconfidence.

Overview With Examples

A confidence interval is a way to say "about" with discipline. It reports an estimate as a range, acknowledging that a sample is not the full truth.

For example, saying pass rate is 92% sounds exact. Saying the approximate 95% confidence interval is 85% to 96% tells the team how much uncertainty remains.

A confidence interval is a way to express uncertainty around an estimate.

Suppose you test 100 outputs and 92 pass. The observed pass rate is 92%. It is tempting to say, "This system passes 92% of the time." But that is too precise. You did not test every possible future output. You tested a sample.

A better report might say, "In our sample, 92% of outputs passed. The approximate 95% confidence interval is 85% to 96%." In plain English, that means the sample suggests the true pass rate is probably somewhere in that range, assuming the sample and test assumptions are reasonable.

The same idea applies to average scores. If 100 outputs have a mean score of 8.1, a confidence interval might estimate the true average as roughly 7.7 to 8.5. The point is not that the interval is magic. The point is that the tester is being honest about uncertainty.

Confidence intervals are especially useful when comparing versions. Imagine Version A has an average score of 8.0 and Version B has an average score of 8.2. Is B really better? Maybe. If the confidence intervals are wide, the difference may be too uncertain to trust. If the interval for the difference is clearly above zero, the evidence is stronger.

Sample size affects interval width. With 20 samples, the interval may be wide because the estimate is uncertain. With 200 samples, the interval usually narrows. The average may stay the same, but your confidence in the estimate improves.

Confidence intervals also help teams avoid overreacting to small samples. If a new prompt scores 9.0 across five examples, the result may look amazing. But the uncertainty is huge. A confidence interval reminds the team that five examples are not enough evidence for a high-risk release.

The language matters. Testers should get comfortable saying "about," "estimated," and "based on this sample." Those words do not weaken the report. They make it more trustworthy.

A confidence interval is a professional way to say: we measured this, here is the estimate, and here is how uncertain we are.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, confidence intervals tell the reader how uncertain the measured sensitivity or specificity is. A model with 94% measured sensitivity on 50 positive cases may still have a wide interval. The report should say whether the lower bound is acceptable for the clinical use case before anyone treats the point estimate as reliable.

Testing/Quality Example

A quality example is comparing two prompts where the new prompt averages 8.2 and the old prompt averages 8.0. If the confidence interval for the improvement is -0.1 to +0.5, the team should be cautious. If it is +0.2 to +0.7, the evidence is stronger.

Expert Notes

At expert level, choose interval methods that match the metric. A pass rate is a proportion and may use Wilson or exact binomial intervals. An average score often uses a t-based or bootstrap interval, especially when the score distribution is not normal.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

This chapter separates two meanings of confidence that sound similar but behave very differently. One comes from a model's self-assessment. The other comes from measured results across many examples.

A useful mental shortcut is this: model confidence is a claim made by the system, while statistical confidence is evidence gathered about the system. Builders should be much more cautious about the first than the second.

Overview With Examples

AI-reported confidence and statistical confidence are different things. An LLM's confidence is a self-assessment of a single judgment. Statistical confidence comes from sample data and observed variation.

For example, a judge may say it is highly confident that one answer deserves an 8. That does not tell you the true average quality of the whole system.

LLMs can report confidence, but that confidence should not be confused with statistical confidence.

An LLM judge might evaluate an answer and say, "Score: 8 out of 10. Confidence: medium. Likely score range: 7 to 9." That can be useful. It tells the tester that the judge found the case somewhat ambiguous or that the score might depend on interpretation.

But that is not a statistical confidence interval. It is a model's self-reported sense of certainty. Models can be overconfident. They can be underconfident. They can sound certain while being wrong. Their confidence may not be calibrated to real-world accuracy unless you test it.

Statistical confidence comes from sample data. If you score 100 outputs and calculate a mean score of 8.2 with a 95% confidence interval from 7.8 to 8.6, that interval is based on observed variation and sample size. It is a calculation, not a self-assessment.

Both forms of confidence can be useful, but they answer different questions. AI-reported confidence helps prioritize review. Low-confidence judgments may deserve human attention because the case is ambiguous. Statistical confidence helps estimate system behavior across a sample.

A good evaluation report labels them clearly. It might say: the LLM judge scored this individual output 8 out of 10 with medium confidence. Across 100 outputs, the observed mean was 8.2 with a 95% confidence interval of 7.8 to 8.6.

The first statement is about one judgment. The second statement is about measured performance across a sample.

Calibration is important. If an LLM judge says it is highly confident on cases where human experts often disagree, that confidence is not very useful. Testers should periodically compare judge confidence and judge scores against human review.

The safest rule is simple: AI confidence is a signal. Statistical confidence is a calculation. Treat them differently, report them differently, and never let model confidence pretend to be proof.

Examples

Web Search Example

Production queries and synthetic edge cases become durable eval assets when they are labeled, versioned, sliced, and tied to the ranking or retrieval failure they expose.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Production conversations and synthetic conversations become durable eval assets when they are anonymized, labeled, clustered, and promoted into regression suites with clear expected behavior.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Production traces become durable eval assets when prompts, repo snapshots, diffs, test outcomes, review comments, and escaped defects are preserved as replayable tasks.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, a model saying it is confident is not the same as calibrated clinical confidence. Test whether high-confidence detections are actually more likely to be correct, whether low-confidence cases are escalated, and whether calibration holds across scanners, hospitals, demographics, and image quality.

Testing/Quality Example

A testing/quality example is reporting both levels separately: the judge marked 12% of individual cases low confidence, and across 300 scored outputs the observed mean was 8.1 with a 95% confidence interval from 7.8 to 8.4.

Expert Notes

Expert teams calibrate AI confidence. They check whether high-confidence judge decisions actually agree with expert humans more often than low-confidence decisions. If not, the confidence label is not useful for routing or release decisions.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

A t-test can sound like heavy statistics, but the builder's intuition is straightforward: when one version looks better, ask whether the gap is big compared with the natural noise in the scores.

The test is not trying to replace judgment. It is a structured way to ask, "Did the new version win by enough, across enough comparable cases, that we should take the improvement seriously?"

Overview With Examples

A t-test helps compare average scores between versions while accounting for variation and sample size. It asks whether an observed average difference is larger than you would expect from noise alone under the test assumptions.

For example, if a new prompt scores 0.5 points higher on average, a t-test helps decide whether that improvement is likely to be real enough to discuss seriously.

When teams improve prompts, models, ranking systems, or AI workflows, they need to know whether the new version is actually better.

Suppose the old prompt has an average score of 7.8 and the new prompt has an average score of 8.3. The new prompt looks better. But a tester should ask: is that difference meaningful, or did the new version happen to get an easier sample?

A t-test helps answer that question for average numeric scores. It considers the difference between the averages, the amount of variation in the scores, and the number of samples. A large difference with low variation and many samples is more convincing than a small difference with high variation and few samples.

T-tests are useful when outputs are scored numerically, such as with a 0-10 quality rubric. They are commonly used to compare an old prompt against a new prompt, one model against another, or one ranking algorithm against an experimental version.

For product testing, a paired t-test is often the best pattern. In a paired setup, you run the same test cases against both versions. Each case gets two scores: old and new. Then you compare the per-case differences.

This matters because some cases are harder than others. If Version A gets a difficult sample and Version B gets an easy sample, a simple comparison may be misleading. Pairing controls for case difficulty. Case 1 might improve from 7 to 8, Case 2 might stay at 9, and Case 3 might improve from 6 to 8. The test evaluates those differences directly.

A t-test is not a release gate by itself. It focuses on average scores. Many product risks live in the tails. A new model may improve average quality while increasing rare policy failures. A prompt may sound better while occasionally leaking sensitive information. A ranking change may improve overall relevance while hurting one important user segment.

That is why t-tests should be used alongside failure rates, confidence intervals, worst-case review, stratified reporting, and hard safety checks.

A good report might say: the new version improved average score by 0.5 points; the paired t-test produced p = 0.02; the 95% confidence interval for the improvement was +0.2 to +0.8; failure rate did not increase; no critical failures were observed.

That gives the team a stronger basis for decision-making than "the new version looked better in a few examples." Use t-tests to compare average quality. Use judgment and risk analysis to decide whether the improvement is safe and worth shipping.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, a t-test may compare average scores, but clinical release often depends on paired case-level outcomes: did the new model catch more true positives without creating too many false positives? Use statistical tests as one signal, then inspect error categories and severe misses.

Testing/Quality Example

A testing/quality example is scoring the same 200 prompts with the old and new prompts, then running a paired t-test on the per-case score differences. The report includes the mean improvement, confidence interval, p-value, and checks for worse tail behavior.

Expert Notes

At expert level, use paired tests when the same cases run through both versions. Pairing reduces noise from case difficulty. Also inspect assumptions: outliers, non-normal differences, multiple comparisons, and category-specific regressions can all make a tidy p-value misleading.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

A p-value is one of the easiest statistics to misuse, so it helps to start gently. Think of it as a surprise meter under a specific assumption.

The assumption is: "What if there were no real difference?" The p-value asks how surprising your observed result would be in that world. It does not tell you whether the product is good, safe, important, or ready to ship.

Overview With Examples

A p-value is evidence about surprise under a null assumption. It is not a probability that the new version is better, and it is not permission to ship.

For example, p = 0.02 says the observed difference would be fairly surprising if there were truly no difference under the test assumptions. It does not say the difference is important.

A p-value helps answer a specific question: if there were no real difference between two versions, how surprising would our observed result be under the test assumptions?

Suppose the old prompt has an average score of 7.8, the new prompt has an average score of 8.3, and the test returns p = 0.02. A tester-friendly interpretation is: if the old and new prompts were actually equal in quality, a difference this large would be fairly unlikely under the assumptions of the test.

That is useful evidence. It suggests the observed difference may not be random noise.

But p-values are easy to misuse. A p-value of 0.02 does not mean there is a 98% chance the new prompt is better. It does not mean the improvement is important. It does not mean the test was well designed. It does not mean the system is safe to ship.

Many teams use p < 0.05 as a threshold for statistical significance. That can be a helpful convention, but it is not a law of nature. A result just below 0.05 is not magically true, and a result just above 0.05 is not automatically useless.

P-values also say nothing about practical importance. With a very large sample, a tiny improvement can produce a small p-value. For example, an average score increasing from 8.10 to 8.12 may be statistically significant. Users may never notice. The improvement may not justify higher latency, higher cost, or increased risk.

A better quality report puts the p-value in context. It includes the effect size, which tells how large the improvement is. It includes a confidence interval, which shows uncertainty around the improvement. It includes failure rates, which show whether bad outputs became more or less common. It includes category breakdowns, because an overall improvement can hide a regression in a high-risk segment.

The best short rule is this: a p-value can tell you whether a difference is surprising; it cannot tell you whether users will care.

Use p-values as evidence, not permission. They belong in the report, but they should not be the headline and they should never replace product judgment.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, a p-value can say the observed difference is unlikely under a null model, but it does not say the change is clinically useful. A tiny statistically significant lift may not justify workflow disruption. A nonsignificant severe-failure reduction may still deserve more data because the stakes are high.

Testing/Quality Example

A quality example is a prompt change with p = 0.01 but an average improvement of only 0.03 points and a new privacy failure. The p-value is real evidence, but the release decision should still be no.

Expert Notes

Expert reports pair p-values with effect sizes, confidence intervals, sample size, assumptions, and practical risk. They also watch for p-hacking, repeated peeking, and multiple comparisons, all of which can create false confidence.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

This distinction is where the math hands the decision back to humans. Statistical significance asks whether a result looks unlikely to be pure noise. Practical significance asks whether anyone should care.

A tiny improvement can be statistically real and still useless. A risky failure reduction can be practically important even before the evidence is perfect. Good testing reports both the measurement and the meaning.

Overview With Examples

Statistical significance asks whether a difference is likely to be more than random noise. Practical significance asks whether the difference matters enough to change a product decision.

For example, a huge sample can make a tiny improvement statistically significant, while users may never notice the change.

Statistical significance and practical significance answer different questions.

Statistical significance asks whether an observed difference is likely to be more than random sampling noise under a particular test. Practical significance asks whether the difference matters to users, the business, or the risk profile.

A result can be statistically significant and still not matter. Imagine an AI assistant improves from an average score of 8.10 to 8.12 across a very large sample, with p = 0.01. The improvement may be statistically credible. But it is tiny. If it increases latency or cost, the team may reasonably decide not to ship it.

A result can also be practically important even if more data is needed. Suppose a new prompt appears to reduce policy failures from 6% to 2%, but the sample is small and the confidence interval is wide. That change could matter a lot, but the team may need more samples before trusting the estimate.

This distinction is especially important in AI systems because averages can distract from risk. A small average improvement may not matter if rare catastrophic failures increase. A modest average improvement may matter greatly if it reduces a high-risk failure category.

Testers should report both the evidence and the impact. The evidence includes metrics such as average score, confidence interval, p-value, sample size, and failure rate. The impact includes user experience, safety, cost, latency, compliance exposure, and business value.

A practical report might say: the score improvement is statistically significant, but the effect size is only +0.02 and latency increased by 30%, so the change is not recommended. Another report might say: the average score improved only slightly, but policy-boundary failures dropped from 6% to 2%, so the change is worth further validation.

The key is to avoid treating statistical significance as a shipping decision. It is an input to the decision. Product context decides whether the change matters.

Good testers help teams understand both questions: is the difference credible, and is the difference important? A strong release recommendation needs both.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, practical significance matters more than leaderboard movement. A 0.3-point quality lift may be irrelevant if it does not improve patient workflow, reduce false negatives, or improve clinician trust. Conversely, a small average change in a rare severe-failure slice may be worth major attention.

Testing/Quality Example

A testing/quality example is a model upgrade that improves average score from 8.10 to 8.14 with p < 0.01 but increases latency by 25%. The math says the difference is detectable. The product decision may still reject it.

Expert Notes

At expert level, define the minimum meaningful effect before testing. If the team only cares about improvements of at least 0.3 points or a 20% reduction in policy failures, say so before looking at the data.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Metamorphic testing checks relationships between outputs instead of requiring one exact answer. It is especially useful when there are many acceptable outputs but some properties must remain stable.

For example, rewriting a user question should not change the underlying refund policy. Translating a prompt should not remove a safety constraint.

Metamorphic testing is one of the most useful techniques for non-deterministic systems.

The idea is simple. Instead of checking one exact output, you change the input in a controlled way and test whether the relationship between outputs still makes sense.

Consider a refund-policy assistant. The user asks, "Can I return shoes after 45 days?" The policy says returns are allowed within 30 days only. The assistant should explain that the return is outside the standard window.

Now paraphrase the input: "I bought shoes a month and a half ago. Can I send them back?" The wording changed, but the policy fact did not. The answer should still preserve the 30-day rule.

That is a metamorphic relationship. The output does not need to be identical. It does need to remain consistent on the property that matters.

This technique is powerful because many AI systems do not have one perfect expected answer. A summary can be phrased many ways. A search result list can vary. A recommendation engine can return different valid items. But important relationships should still hold.

For summarization, adding an irrelevant sentence to the source should not dramatically change the main summary. For ranking, adding a clearly worse candidate should not cause the best candidate to disappear from the top results. For classification, changing a customer's name should not change a risk classification unless name is a valid and intended signal.

Metamorphic testing can also expose brittleness. If small typos, paraphrases, or irrelevant details cause large changes in factual answers, the system may be unreliable. If translating a policy question into another language changes the business rule, the multilingual behavior needs attention.

Useful transformations include paraphrasing, changing names, adding irrelevant details, reordering facts, shortening or lengthening the prompt, translating the input, and introducing realistic typos. Each transformation should have a clear expectation. The tester should know which property should remain stable and which variation is acceptable.

The strength of metamorphic testing is that it lets testers evaluate consistency without requiring a single golden output. It is a natural fit for LLMs, recommendation systems, ranking systems, search, classifiers, and AI agents.

When exact answers are hard to define, test the relationships that must remain true.

Examples

Web Search Example

A metamorphic test might say that adding a city name should shift local results, while fixing a typo should preserve the user's intent rather than destroy relevance.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A metamorphic test might say that rephrasing a question should preserve the answer, while adding a safety-sensitive detail should change the refusal or escalation behavior.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

A metamorphic test might say that renaming a variable should not change behavior, while changing an API contract should force corresponding tests, docs, and call sites to change.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is taking a support question and creating variants: polite wording, angry wording, shorter wording, translated wording, and typo-heavy wording. The answer may vary, but the policy fact and safe next step must remain consistent.

Expert Notes

Expert metamorphic suites define relation types explicitly: invariance, monotonicity, symmetry, subset consistency, ranking stability, or conservation of key facts. Each relation should have a clear oracle for what must remain true after transformation.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Golden sets and live sampling answer different questions. Golden sets preserve known important cases. Live sampling discovers what is happening now.

For example, a golden set might contain past policy failures, while live sampling captures this week's new user questions and emerging abuse patterns.

A golden set is a curated collection of important test cases. It usually includes known edge cases, previous failures, high-risk policy boundaries, common user questions, and examples reviewed by domain experts.

Golden sets are valuable because they preserve hard-won knowledge. When a model once failed a refund boundary, leaked a sensitive field, or misunderstood a legal disclaimer, that example should not disappear from the test strategy. It should become part of the regression suite.

But golden sets have a weakness. They can get stale. Products change. Policies change. Users change. Abuse patterns change. A test set that represented reality six months ago may slowly become less useful. A system can also overfit to a golden set, performing well on familiar examples while failing on fresh ones.

Live sampling addresses that weakness. It uses recent production-like inputs, current support conversations, new edge cases, and real user behavior. Live samples reveal what is happening now, not only what the team already knows to worry about.

The two approaches answer different questions. Golden sets ask, "Did we regress on important known cases?" Live sampling asks, "How are we doing on current reality?"

A mature testing strategy uses both. Before release, run the golden set to catch regressions. During evaluation, sample realistic new cases to estimate current quality. After release, keep sampling production behavior to detect drift. When live sampling finds an important new failure, add it back into the golden set.

This creates a learning loop. The golden set becomes the team's memory. Live sampling becomes the team's contact with reality.

For LLM systems, this is especially important because the product surface changes quickly. Users discover new ways to ask questions. Attackers discover new prompt injection patterns. Retrieval data changes. Model versions change. Static tests alone cannot keep up.

The practical rule is simple: golden sets catch regressions; live sampling catches drift. Do not choose between them. Use both, and let each one improve the other.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is running a 300-case golden set before release and a 100-case live sample every day after release. New serious production failures are added back to the golden set so the suite learns from reality.

Expert Notes

At expert level, golden sets should be versioned, deduplicated, labeled by risk, and periodically refreshed. Live samples should preserve privacy and represent the current traffic mix instead of only the cases that are easiest to review.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Risk-based sampling puts more measurement effort where failure hurts more. Equal sampling feels tidy, but users experience consequences, not test-plan symmetry.

For example, billing, privacy, account deletion, medical advice boundaries, and policy enforcement deserve deeper sampling than low-risk style variations.

Not every feature deserves the same amount of testing.

A casual AI writing assistant that suggests alternate phrasing has a different risk profile from an AI agent that can cancel subscriptions, issue refunds, or answer medical questions. Treating those systems the same is not efficient and it is not safe.

Risk-based sampling means allocating more test effort to the areas where failure would hurt most. The sample size, review depth, and release gate should reflect the impact of a bad outcome.

High-risk areas include payments, refunds, account deletion, medical content, legal content, financial advice, privacy-sensitive flows, security-sensitive flows, policy boundaries, and irreversible actions. A mistake in these areas can harm users, create compliance exposure, or damage trust.

Risk has several dimensions. Impact asks how bad the failure would be. Likelihood asks how often it might happen. Detectability asks whether users or systems would notice quickly. Reversibility asks whether the action can be undone. Regulatory exposure asks whether the mistake creates legal or compliance risk.

A low-risk feature may need a smaller sample and lighter gates. A medium-risk feature may need category-level reporting and manual review of low scores. A high-risk feature may need a larger sample, strict hard-failure rules, human review of boundary cases, adversarial testing, and post-release monitoring.

This does not mean testers ignore low-risk areas. It means they spend measurement effort where uncertainty is most expensive.

Risk-based sampling also helps teams communicate priorities. Instead of saying, "We tested everything equally," a tester can say, "We used more samples for billing, account actions, and policy boundaries because failures there are less reversible and more harmful." That is a stronger quality argument.

Equal sampling may feel fair, but it is usually not the right strategy. Testing should follow risk because users experience the consequences, not the test plan.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is allocating 50 cases to low-risk FAQ answers, 200 cases to billing and refunds, 200 cases to policy edge cases, and targeted adversarial tests for privacy leakage. The sample reflects impact.

Expert Notes

Expert risk sampling combines likelihood, severity, detectability, reversibility, and exposure. A rare failure with irreversible harm may deserve more testing than a frequent cosmetic issue.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Stratified reporting breaks results into meaningful categories so weak spots do not hide inside a good average. It shows where quality is strong and where risk clusters.

For example, an assistant may average 8.4 overall but score 6.1 on Spanish billing questions or 5.8 on account deletion cases.

One average can hide the most important quality problem.

Suppose an AI support assistant has an overall average score of 8.4. That sounds good. But the breakdown tells a different story: English support cases score 8.8, Spanish support cases score 6.9, billing cases score 8.6, and policy edge cases score 5.8.

The overall average was not false. It was incomplete.

Stratified reporting means breaking results into meaningful categories. Those categories might include language, persona, input type, product area, risk level, customer segment, prompt length, device type, geography, policy category, or new versus returning users.

The right categories depend on the product. For an LLM support assistant, language and policy category may matter. For a recommendation engine, product category and customer segment may matter. For a fraud model, geography and transaction type may matter. For an AI agent, action type and reversibility may matter.

Stratified reporting is important because non-deterministic systems often perform unevenly. A model may be excellent with short English prompts and weak with long multilingual prompts. A ranking system may work well for popular inventory and poorly for rare items. An agent may answer questions safely but behave poorly when allowed to take actions.

A good report starts with the overall result, then immediately shows the most important breakdowns. It might say: overall average score is 8.4 and failure rate is 3.2%, but policy edge cases average 5.8 with an 18% failure rate. Recommendation: do not ship until policy-boundary behavior improves.

This kind of reporting prevents teams from hiding behind a comfortable average. It also helps engineering teams focus. Instead of "quality is bad," the report says, "quality is good overall, but Spanish policy edge cases are failing." That is actionable.

For high-risk systems, category-specific gates may be necessary. The overall score may need to be at least 8.0, but billing, privacy, and policy categories may each need their own thresholds.

Users do not experience the average. They experience their segment, their language, their workflow, and their edge case. Stratified reporting makes those experiences visible.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is reporting results by language, product area, risk level, prompt length, and policy category. The release gate can require acceptable overall quality and acceptable quality in every high-risk stratum.

Expert Notes

At expert level, define strata before the evaluation and ensure each important stratum has enough samples to support a decision. Too many tiny categories create noisy numbers; too few categories hide actionable risk.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Rare failures matter when severity is high. A system that behaves well 99% of the time can still be unshippable if the remaining 1% includes privacy leaks, unsafe instructions, or irreversible actions.

For example, an AI agent that usually books the right trip but occasionally confirms without approval has a rare-failure problem, not a small average-quality problem.

Some failures are rare but unacceptable.

An AI system can produce excellent answers 99% of the time and still occasionally leak private data, hallucinate a dangerous instruction, approve an invalid refund, or violate a safety policy. If the failure is severe enough, the high average does not make the system shippable.

This is why testers need rare failure hunting.

Averages are not designed to protect against catastrophic tails. Imagine 99 outputs score 9 and one output scores 0. The average score is 8.91. That sounds excellent. But if the score of 0 represents a privacy leak or unsafe instruction, the release risk is still real.

Rare failures require different tactics. Larger random samples can help, but they are not enough by themselves. If a failure is very rare, random sampling may miss it. Targeted stress testing is necessary.

For LLM systems, rare failure hunting may include prompt injection attempts, jailbreak attempts, policy-boundary questions, requests involving private data, ambiguous instructions, malicious users, conflicting instructions, and multilingual edge cases. For AI agents, it should include irreversible actions, tool misuse, permission boundaries, and cases where the agent should ask for confirmation or escalate.

The metrics should also change. Average score is not enough. Track the worst observed output, the number of critical failures, the percentage below a threshold, safety failure rate, privacy failure rate, and policy violation rate. Include examples of the worst outputs in the report so decision-makers can see the risk directly.

Rare failure hunting also belongs after release. Some failures only appear under real traffic, real user creativity, or real abuse pressure. Production monitoring, complaint analysis, and sampled evaluations can reveal failures the lab missed.

The lesson is blunt: average quality tells you how the system usually behaves. Rare failure hunting tells you whether the system can be trusted when it does not.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is combining random samples with targeted attack suites: jailbreak attempts, boundary policy cases, ambiguous user intent, missing context, malicious documents, and repeated runs of historically brittle prompts.

Expert Notes

Expert rare-failure work treats zero observed failures carefully. If you test 100 cases and see zero failures, you have evidence, not proof. The upper bound on the plausible failure rate may still be too high for safety-critical behavior.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Pairwise comparison asks which of two outputs is better. It is often easier and more reliable than asking for an absolute score, especially when quality is nuanced.

For example, reviewers may argue whether an answer is a 7 or 8, but agree that version B is clearer, safer, and more faithful than version A.

Sometimes it is easier to compare two outputs than to assign each one an exact score.

That is the idea behind pairwise comparison. You run the same input through two versions, such as an old prompt and a new prompt. Then a human or LLM judge decides which output is better according to the rubric: A, B, or tie.

After many cases, you calculate a win rate. For example, the new version may be preferred in 68% of cases, the old version in 21%, with 11% ties. That tells a clear story about preference.

Pairwise comparison is useful because absolute scoring can be difficult. Reviewers may disagree about whether an answer is a 7 or an 8. But they may agree that one answer is more correct, more complete, safer, or more useful than another.

This approach works well for prompt changes, model upgrades, ranking changes, summarization quality, writing quality, and assistant responses. It is especially helpful when the goal is to compare versions rather than certify an absolute level of quality.

The judge still needs a rubric. "Better" should not mean "longer" or "more confident." It should mean better according to product goals: more correct, more faithful, more helpful, safer, clearer, or more aligned with policy.

Bias control matters. Randomize whether the old or new output appears first. Hide version labels. Allow ties when neither output is clearly better. Review disagreement cases, especially when the judge chooses a fluent but factually weaker answer.

Pairwise comparison should not replace hard safety checks. A new answer may be better than the old one and still unacceptable. If both outputs violate policy, choosing the better one is not enough. The report should still track critical failures, failure rates, and category-level performance.

Used well, pairwise comparison gives testers another practical measurement tool. It helps answer the question teams often care about most: did this change make the product better than what we had before?

Examples

Web Search Example

This concept applies to query understanding, ranking, retrieval, snippets, safety, latency, and result satisfaction. The test should ask how pairwise comparison changes what users see across realistic query slices.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

This concept applies to conversation quality, grounding, tone, refusal, memory, tool use, escalation, and recovery. The test should ask how pairwise comparison changes the user's outcome across realistic conversations.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

This concept applies to task understanding, file selection, code edits, tests, tool use, reviewability, security, and maintainability. The test should ask how pairwise comparison changes the quality of generated patches across realistic coding tasks.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is comparing old and new prompts on the same 300 inputs. Reviewers see outputs A and B in randomized order, choose A, B, or tie, and label the reason: correctness, completeness, safety, clarity, or policy fit.

Expert Notes

At expert level, blind the version labels, randomize side order, allow ties, and analyze win rate by category. Pairwise wins do not replace absolute gates because the better of two bad outputs can still be unacceptable.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Reproducibility in non-deterministic systems is less about forcing the exact same output and more about preserving enough context to explain and investigate the failure.

For example, an LLM failure may depend on model version, retrieved documents, tool outputs, prompt configuration, temperature, timestamp, or feature flags.

Non-deterministic failures can be frustrating because they may not reproduce on demand. A tester sees a bad output. An engineer tries the same input and gets a good output. Without logs, the team is left guessing.

That is why reproducibility starts with capturing context.

For LLM systems, the important context includes the user input, system prompt, developer prompt, model name, model version, temperature, seed if available, retrieved documents, tool calls, tool outputs, timestamps, feature flags, configuration, final output, judge score, and judge explanation.

Each item helps answer a different question. The prompt shows what instructions the model received. The model version shows whether behavior changed because of an upgrade. Retrieved documents show what evidence the model saw. Tool outputs show whether the model acted on bad data. The judge explanation shows why the output was considered a failure.

For distributed systems, context may include request IDs, event IDs, service versions, timing, retries, cache state, region, feature flags, and dependency responses. The goal is the same: reconstruct the conditions around the failure.

Perfect reproduction is not always possible. Even with all the logs, an LLM may not produce the exact same answer again. That is normal. The goal is not always exact replay. The goal is to make the failure explainable, diagnosable, and testable again.

Good logs also improve evaluation quality. If a judge gives a low score, the team can inspect the input, output, policy context, and explanation. If a failure becomes part of the golden set, the captured context helps preserve the lesson.

Without logs, failures become arguments. The tester says it failed. The developer cannot reproduce it. The team debates whether it was real. With logs, failures become evidence.

For non-deterministic systems, logging is not an afterthought. It is part of the test design. If you cannot replay or explain the conditions around a failure, you can barely debug it.

Examples

Web Search Example

Log the query, locale, time, index version, ranking model, filters, retrieved candidates, final ranking, latency, and clicked or judged outcomes.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Log the prompt, system message, model, retrieved context, tool calls, intermediate state, final answer, judge score, cost, latency, and any human escalation.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Log the prompt, repo state, files read, commands run, tool calls, diffs, tests attempted, failures observed, model version, cost, latency, and reviewer outcome.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is logging the user input, final output, judge score, judge explanation, system prompt version, model version, retrieval IDs, tool calls, feature flags, seed if available, and request ID whenever an evaluation fails.

Expert Notes

Expert logging distinguishes replay data from diagnosis data. Replay data tries to recreate conditions. Diagnosis data explains why the system behaved that way. Both should be privacy-aware, access-controlled, and tied to durable artifact IDs.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

LLM judges need calibration because they are evaluators, not truth machines. Calibration checks whether the judge scores like trusted human reviewers on the cases that matter.

For example, a judge may reward polished writing while missing a subtle policy contradiction. Calibration makes that weakness visible before the judge is used at scale.

LLM judges can make large-scale evaluation possible, but they need calibration.

The central question is simple: does the judge score outputs the way our best human testers would?

The answer can be yes, at least in some settings. The Berkeley/LMSYS "Judging LLM-as-a-Judge" work showed GPT-4 reaching over 80% agreement with human preferences on MT-Bench and Chatbot Arena evaluations, which was roughly comparable to human-human agreement. That is a big deal. It means LLM judges are not just a toy; they can approximate human preference well enough to be operationally useful.

But that result is not universal permission. A judge that tracks human preference on chatbot comparisons may still fail on your legal policy, medical summary, internal coding standard, safety rule, or domain-specific support workflow.

Without calibration, an LLM judge may be too lenient, too harsh, inconsistent, biased toward fluent writing, weak on domain-specific policy, or overly impressed by confident language. If you scale an uncalibrated judge, you may scale the wrong judgment.

A calibration process starts with a sample of outputs. Human reviewers score those outputs using the same rubric as the LLM judge. Then the team compares human scores with judge scores. Where did they agree? Where did they disagree? Were the disagreements random, or did they reveal a pattern?

For example, the LLM judge may give high scores to answers that are polite and well written but subtly contradict policy. That suggests the rubric or judge prompt needs to emphasize policy compliance more strongly. The judge may over-penalize brief answers even when they are correct. That suggests the rubric should clarify when concision is acceptable.

Disagreement can also reveal that the humans need alignment. If expert reviewers disagree often, the problem may not be the judge. The rubric may be vague, the policy may be ambiguous, or the examples may be genuinely borderline.

Calibration improves when rubrics include examples. Show what a 10 looks like. Show what a 7 looks like. Show what a 3 looks like. Show hard failures that should receive very low scores regardless of tone. Concrete examples help both humans and LLM judges apply the criteria more consistently.

Calibration should not be a one-time event. Model behavior changes, product policy changes, and evaluation needs change. Periodic calibration keeps the judge aligned with the product's current definition of quality.

The goal is not perfect agreement. The goal is to know where the judge is reliable, where it is weak, and which cases need human escalation.

LLM judges are powerful when calibrated. Treat judge quality as something you test, not something you assume.

Examples

Web Search Example

An LLM judge can review a query and result list, score whether the top results satisfy intent, and explain why a result is irrelevant, stale, spammy, or unsafe.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

An LLM judge can score an answer against a rubric, compare two candidate responses, identify unsupported claims, and flag tone or policy problems for human review.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An LLM judge can review a diff, summarize risk, spot likely missing tests, compare approaches, and flag suspicious code, but it still needs executable checks and human calibration.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is selecting 150 outputs, having two expert humans and the LLM judge score each one, then reviewing disagreements. The team updates the rubric, adds examples, and defines which cases require human escalation.

Expert Notes

At expert level, track judge-human agreement over time, by category, and by severity. A judge can be acceptable for low-risk style checks and unacceptable for regulated policy decisions. Calibration should produce routing rules, not just a single accuracy number.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Inter-rater agreement measures whether reviewers apply the evaluation criteria consistently. Reviewers can be humans, LLM judges, or both.

For example, if one reviewer scores an answer 9 and another scores it 3, the issue may be the output, the rubric, the policy, or reviewer training.

Inter-rater agreement measures whether multiple reviewers agree. The reviewers may be humans, LLM judges, or a combination of both.

Agreement matters because evaluation is only useful if the criteria can be applied consistently. If one reviewer scores an answer 9 and another scores it 3, the team needs to understand why.

Disagreement can mean several things. The rubric may be vague. The product policy may be unclear. The output may be genuinely borderline. One reviewer may be too strict. Another may be too lenient. An LLM judge may be biased toward confident writing or may miss domain-specific details.

You do not need advanced statistics to begin tracking agreement. Start simply. Have three reviewers score 100 outputs. Count how often all reviewers agree, how often two agree, and how often all disagree. Then review the disagreement cases.

The value is in the discussion. If reviewers disagree because the policy is unclear, the product team may need to clarify the policy. If they disagree because the rubric does not define "complete" or "safe" precisely enough, the rubric needs improvement. If they disagree only on borderline cases, those cases may need escalation rules.

More advanced teams can use measures such as Cohen's kappa or Krippendorff's alpha. These statistics adjust for agreement that might happen by chance. They can be useful, especially when evaluation becomes part of a formal release process. But they are not required to get started.

Inter-rater agreement also helps calibrate LLM judges. If the LLM judge agrees with expert humans on clear cases but struggles on ambiguous policy boundaries, testers know where to add human review.

The key insight is that disagreement is not just noise. It is information. It tells you where the evaluation system is unclear, where the product behavior is ambiguous, and where automated judgment may be risky.

If good reviewers cannot agree, the system may not be the only thing that needs fixing. The definition of quality may need work too.

Examples

Web Search Example

Raters can judge query-result relevance, freshness, spam, trustworthiness, and whether the result satisfies the likely intent. Disagreement often reveals ambiguous intent or weak guidelines.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Raters can judge correctness, helpfulness, policy compliance, empathy, grounding, and escalation quality. Disagreement often reveals vague rubrics or missing examples.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Reviewers can judge whether a patch is correct, minimal, idiomatic, secure, tested, and easy to maintain. Disagreement often reveals unclear engineering standards or hidden product intent.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is having three reviewers score the same 100 outputs, then tracking exact agreement, within-one-point agreement, and disagreement on hard-failure labels. The disagreement cases become rubric-improvement material.

Expert Notes

Expert teams use agreement statistics carefully. Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha adjust for chance agreement, but they still depend on label design, prevalence, reviewer training, and whether the task is ordinal or categorical.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

A release gate is where math becomes an operational promise. The numbers are not there to decorate the report; they define how much uncertainty, failure, cost, and risk the team is willing to accept.

Before choosing thresholds, explain the human reason for each one. A lower confidence bound protects against overclaiming. A severe-failure blocker protects trust. A latency threshold protects the user experience. The math should serve those decisions.

Overview With Examples

A release gate for non-deterministic systems should combine average quality, uncertainty, tail risk, hard failures, and category-level results. One number is not enough.

For example, a model can improve average quality while introducing a rare privacy leak. A good gate catches both the improvement and the new blocker.

Non-deterministic systems need release gates that reflect uncertainty.

A traditional gate might say all tests must pass. That is still useful for deterministic checks, but it is not enough for AI systems, ranking systems, recommendation engines, or agents whose behavior varies.

A stronger release gate combines several kinds of evidence. It might require an average score of at least 8.0, a lower bound of the 95% confidence interval above 7.5, at least 95% of outputs scoring 7 or higher, a policy failure rate below 2%, no output below 4, no critical safety or privacy failures, and latency increase below 10%.

Each part protects against a different risk. The average score measures overall quality. The confidence interval prevents overclaiming from noisy samples. The percentage above threshold checks consistency. The policy failure rate tracks a specific business risk. The minimum score protects against terrible outliers. The critical-failure rule protects safety and trust. Latency and cost checks prevent quality improvements from hiding operational regressions.

The gate should match the product risk. A low-risk creative assistant may use lighter thresholds. A billing agent, medical assistant, legal tool, or account-action agent should use stricter gates and larger samples.

Category-specific gates are often necessary. An overall score of 8.4 may hide a score of 5.8 on policy edge cases. A release gate can require both overall quality and acceptable performance in high-risk categories.

Hard failures should remain hard. If the system leaks private data, executes an unsafe action, or contradicts a regulated policy, it should not pass because the average score is high.

A release gate should also state what happens after release. Non-deterministic systems can drift. A good gate may include canary rollout, shadow testing, production sampling, rollback triggers, and monitoring thresholds.

The purpose of a release gate is not to create a false sense of certainty. It is to define what level of evidence and risk the team considers acceptable.

A good gate says: the system is good enough on average, the uncertainty is understood, the important categories are safe enough, and the worst observed failures are controlled.

Examples

Web Search Example

A web search engine may return slightly different rankings as indexes refresh, personalization changes, ads rotate, or ranking features update. The test is whether the most useful content for the intent still rises to the top, not whether every result appears in the same slot forever.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A chatbot may answer the same question twice, in different words, with the same meaning and impact. The test is whether the answer remains correct, grounded, safe, and useful, not whether the sentence is identical.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An AI coding agent may solve the same ticket with different edits, helper functions, or file boundaries. The test is whether the behavior, maintainability, and safety hold, not whether the patch looks identical.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, release gates should include minimum sensitivity, maximum false-positive rate, calibration thresholds, subgroup performance, expert-review agreement, and a clear escalation path. A model should not ship because the average score improved if performance regressed for a protected population or a critical pathology.

Humanoid Robot Example

For humanoid robots and embodied AI, release gates should include safety-envelope violations, near-miss counts, emergency-stop behavior, human-proximity rules, fall risk, force limits, and recovery after sensor errors. A robot should not graduate to a wider rollout because the average task completion rate looks good.

Testing/Quality Example

A testing/quality example is requiring average score >= 8.0, lower confidence-bound >= 7.5, policy failure rate < 2%, no critical failures, p95 latency within target, and high-risk categories above their own thresholds.

Expert Notes

At expert level, release gates should define data freshness, sample composition, minimum sample size, confidence method, severity taxonomy, override process, rollback trigger, and post-release monitoring window.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

For non-deterministic systems, launch is the beginning of real-world measurement. Production behavior changes as users, data, policies, dependencies, and models change.

For example, a support assistant can pass pre-release tests and then drift when the policy database changes or users discover a new edge case.

Testing non-deterministic systems does not stop at launch.

Pre-release testing is necessary, but it cannot cover every future condition. User traffic changes. Policies change. Retrieval data changes. Model versions change. Abuse patterns change. Personalization signals drift. External dependencies behave differently. The system that passed last week may behave differently next month.

That is why AI quality is monitored, not merely certified.

Post-release monitoring should track sampled output quality, failure rates, safety violations, privacy issues, policy violations, user complaints, human escalations, latency, cost, category-level performance, and drift from previous baselines.

Canary releases are one useful pattern. A small percentage of traffic sees the new version first. Testers and engineers monitor quality before expanding rollout. If failure rates increase, the team can stop or roll back before most users are affected.

Shadow testing is another pattern. The new version runs in parallel with the current version, but users do not see its output. The team compares the hidden output against the production output using judges, metrics, and human review. This is especially useful when the new version might be risky but the team wants evidence from realistic traffic.

Rollback thresholds should be defined before release. Examples include: any critical safety failure, policy failure rate above 3%, average quality below 7.5, user complaint rate doubling, or latency increasing more than 25%. Predefined thresholds reduce hesitation when the system starts misbehaving.

Monitoring should also feed the test suite. Important production failures should become new golden-set cases. New abuse patterns should become adversarial tests. New user behaviors should inform future sampling.

The mindset shift is important. For deterministic systems, teams often think of release as the moment testing ends. For non-deterministic systems, release is the moment testing meets reality.

The goal is not to be surprised less because you guessed every possible case. The goal is to build an evaluation loop that notices when reality changes.

Examples

Web Search Example

Release gates should watch relevance by query slice, zero-result rates, unsafe-result rates, latency, click satisfaction, freshness, and regressions on known important queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Release gates should watch severe answer failures, privacy mistakes, unsupported claims, over-refusals, tool-call errors, escalation quality, latency, and cost per resolved conversation.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Release gates should watch build failures, test regressions, security findings, review rejection rate, escaped defects, over-broad diffs, and whether rollback or revert paths are clean.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, monitoring after release should track scanner drift, population drift, label drift, clinician override rates, false alarms, missed-case reviews, and changes in prevalence. A model can be valid at launch and become unreliable when devices, protocols, or patient mix change.

Humanoid Robot Example

For humanoid robots and embodied AI, monitoring after release should capture near misses, operator overrides, unexpected contacts, navigation failures, hardware degradation, and environmental drift. Physical systems age and environments change, so safety evidence must continue after launch.

Testing/Quality Example

A quality example is sampling production outputs daily, scoring them with a judge, auditing high-risk cases with humans, tracking failure rates by category, and triggering rollback when critical failures or threshold breaches occur.

Expert Notes

Expert monitoring separates data drift, model drift, behavior drift, and evaluation drift. If the judge changes, apparent product quality can change even when the product did not. Version every evaluator and baseline.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Non-deterministic testing does not mean every system should be as random as possible. In many products, variation is the point: a chatbot can phrase an answer naturally, a search engine can adapt to freshness and context, and a coding agent can choose a different implementation path. But there are also moments when you want less variation.

You might want determinism in the product itself because the user needs a stable answer, a regulated workflow needs repeatability, a tool call must produce a predictable schema, or a generated report must not drift between runs. You might want determinism in the validation system because you are trying to reproduce a failure, compare two versions, reduce sample size, debug a judge, or keep a release gate from wobbling for reasons that have nothing to do with the product change.

The practical goal is not always perfect determinism. The goal is often lower variance. If the same input produces a narrower range of acceptable outputs, you need fewer samples to understand what changed. If your judge gives more stable scores, your release report becomes less noisy. If your agent chooses tools more consistently, debugging becomes less of a fog machine.

Most people do not know that LLM APIs often expose controls that change how much randomness is used when choosing the next token. The most famous control is temperature. Temperature changes how strongly the model prefers the highest-probability next token. A low temperature makes the model more conservative. A high temperature makes it more willing to sample from lower-probability choices.

At temperature 0, many systems try to use greedy decoding: pick the most likely next token at each step. This often makes output much more repeatable. It does not guarantee identical output across every provider, model, hardware path, tool call, or backend update, but it is one of the first settings to try when you need stability.

Another common control is top-p, also called nucleus sampling. Instead of considering every possible next token, the system keeps the smallest set of tokens whose cumulative probability reaches p. A top_p value of 0.9 means the model samples only from the most likely tokens that together account for 90% of the probability mass. Lowering top_p removes more of the long tail.

Top-k is similar but simpler. It keeps only the k most likely tokens. If top_k is 1, the model can only choose the single most likely token. If top_k is 40, it can choose among the top forty. Top-k is not exposed by every commercial API, but it is common in local and open-model serving stacks.

Some systems also support a seed. A seed controls the pseudo-random number generator used during sampling. If the model version, prompt, parameters, system messages, tools, retrieval context, and backend behavior are unchanged, a fixed seed can make outputs easier to replay. Seeds are useful for debugging and regression investigation, but they should not be treated as a permanent contract unless the provider explicitly promises that.

Other controls can also reduce variation. JSON schema or structured-output modes restrict the shape of the answer. Stop sequences end generation at known boundaries. Logit bias can boost or suppress specific tokens. Repetition, frequency, and presence penalties change how likely the model is to repeat or introduce tokens. These controls are not always about determinism directly, but they narrow or reshape the path the model can take.

The catch is that every determinism control has tradeoffs. Lower temperature may make answers more stable but less creative. A very low top_p can remove useful alternatives. A tiny top_k can make answers brittle. Strict schemas can improve parsing but hide partial uncertainty. Logit bias can force awkward wording. A deterministic judge can be consistently wrong.

So the testing question is not, "Can we make this perfectly deterministic?" The better question is, "Where does determinism help the user, the release decision, or the debugging loop, and where does variation represent useful product behavior?"

Running Examples

Web Search Example

For web search, determinism can help when validating a ranking experiment. Freeze the query set, index snapshot, location, language, personalization state, ads treatment, freshness window, and judge rubric before comparing versions. If those inputs keep changing while the ranking algorithm changes, the measurement system will not know which difference caused the score movement.

You might still allow dynamic freshness for production, but use a frozen slice for regression testing. The deterministic slice answers, "Did the ranker get better on the same evidence?" The live slice answers, "Is the system still good under current internet conditions?"

Chatbot Example

For a chatbot, set temperature low when testing policy compliance, structured output, refusal boundaries, and reproducibility. If the same policy question sometimes produces three different policy interpretations, the team may need either a better prompt or a more deterministic serving configuration.

For creative or conversational features, do not turn everything into a robot. Instead, separate stable requirements from acceptable variation. The assistant may answer the same question twice, in different words, with the same meaning and impact. That is acceptable. Contradicting policy is not.

AI Coding Agent Example

For an AI coding agent, determinism is useful when reproducing a failed patch, comparing model versions, or validating a tool workflow. Pin the model, prompt, repository state, dependencies, environment variables, tool permissions, and seed if available. Also log every command, file read, file write, test run, and error.

Even then, agent workflows may vary because tool outputs, package registries, timestamps, network calls, and hidden model updates can change. Determinism for agents is usually a discipline of versioning and replay, not just a temperature setting.

Testing/Quality Example

A useful quality practice is to run the same eval suite twice with the same model and configuration before testing a product change. If the scores move a lot with no intended change, your measurement system has too much variance. Turn down judge temperature, freeze retrieval context, pin model versions, use structured outputs, or increase samples before trusting release decisions.

Then run a second pass with production-like variation turned back on. The deterministic pass helps debug. The realistic pass helps estimate what users will experience.

Expert Notes

Temperature, top_p, and top_k all affect sampling from the model's next-token probability distribution. They are usually applied after the model computes logits and before the next token is sampled. In many implementations, temperature rescales logits, top_p truncates by cumulative probability, and top_k truncates by rank. Different providers may apply these controls in different orders or expose only some of them.

For strict reproducibility, log the full evaluation envelope: model identifier, model version if available, prompt, system message, tool definitions, retrieval index version, source documents, decoding parameters, seed, schema, judge prompt, judge model, timestamps, and infrastructure version. If any of those change, "same test" may no longer mean same test.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

The new AI quality skillset combines product judgment, evaluation design, sampling strategy, AI-assisted review, risk analysis, and statistical storytelling. The work becomes more strategic because the systems are less predictable.

For example, a developer may design a rubric in the morning, calibrate an LLM judge at noon, analyze confidence intervals in the afternoon, and explain a release recommendation to leadership by the end of the day.

The next generation AI builder does not only check whether one output matched one expectation. They evaluate behavior under uncertainty.

That requires a broader skillset.

The builder becomes a rubric designer. Non-deterministic systems need clear definitions of quality. The builder helps define what correctness, safety, completeness, tone, usefulness, policy compliance, reliability, and fairness mean for the product.

The builder becomes a sampling strategist. They decide which cases matter, how many samples are enough, which categories deserve deeper coverage, and which failures are rare but dangerous. Sampling is not administrative overhead. It is the foundation of credible evidence.

The builder becomes an AI judge operator. LLMs can help evaluate outputs at scale, but developers and quality specialists must write judge prompts, calibrate judge behavior, review disagreements, detect bias, and decide which cases require human escalation.

The tester becomes a risk analyst. They know that average quality is not enough. They watch the tails. They ask whether the system leaks data, violates policy, harms vulnerable users, takes irreversible actions, or fails in high-impact categories.

The tester becomes a statistical storyteller. They explain average scores, failure rates, confidence intervals, p-values, effect sizes, category breakdowns, and uncertainty in language the team can use. They do not hide behind math, and they do not ignore it. They translate evidence into a responsible recommendation.

This skillset makes developers and quality specialists more important, not less. AI does not remove the need for human judgment. It increases the need for people who can define quality, measure uncertainty, and explain risk.

A developer in this world does not say, "It passed once." They say, "Here is how often it behaved acceptably. Here is how bad the failures were. Here is how confident we are. Here is where the risk remains. Here is my ship recommendation."

That is the quality conversation modern AI products need.

The future of testing is not about pretending non-determinism can be forced into old patterns. It is about building new patterns that make unpredictable systems measurable, debuggable, and trustworthy enough to use.

Examples

Web Search Example

A search-quality skill can tell an agent how to build query sets, judge relevance, evaluate freshness, and report NDCG. Test whether the agent follows that workflow when asked to evaluate search quality.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A chatbot-quality skill can tell an agent how to build conversation cases, check grounding, score tone, and test refusals. Test whether the agent uses that rubric instead of inventing a generic checklist.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

A coding skill can tell the agent how to inspect the repo, make scoped edits, run tests, and avoid destructive commands. Test whether it follows that workflow on realistic coding tasks.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is a developer leading an AI release review: defining the sample, setting the rubric, reviewing judge calibration, checking tail failures, explaining uncertainty, and recommending ship, hold, canary, or rollback.

Expert Notes

At expert level, the strongest AI builders become evaluation architects. They design systems that continuously measure quality, generate useful failure evidence, improve test assets from production learning, and make uncertainty understandable to non-statisticians.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Rubrics are the operating system of non-deterministic testing. They tell humans, LLM judges, and product teams what quality means before anyone starts arguing about individual outputs.

For example, a support assistant rubric might evaluate policy correctness, completeness, tone, user actionability, and safety. A medical summary rubric should weight factual accuracy and omission risk far more heavily than polish.

A useful rubric does not simply say "good" or "bad." It defines dimensions. It explains what each score means. It separates hard failures from softer quality problems. It includes examples that help reviewers apply the same standard.

The biggest mistake is building a rubric that sounds impressive but cannot be applied consistently. If one reviewer thinks "complete" means every detail and another thinks it means enough to help the user, the scores will look quantitative while hiding disagreement.

Strong rubrics use anchors. A 10 is not just excellent. It is correct, complete, safe, clear, and ready to ship. A 7 is useful but missing a minor detail. A 4 is weak or risky. A 0 is a hard failure such as fabricated policy, unsafe advice, or private data leakage.

Rubrics should also define blockers. If an answer leaks personal data, it should not pass because it is polite. If an agent executes an irreversible action without permission, the quality score is not the main story. The blocker is.

The rubric should be product-specific. The dimensions for search relevance, customer support, medical summarization, code generation, and autonomous agents are different. Reusing a generic rubric across all of them is convenient and usually wrong.

Rubrics improve over time. Disagreement cases, production failures, and examples that confuse reviewers should feed back into the rubric. A living rubric is a quality asset, not a one-time document.

Examples

Web Search Example

A good rubric separates relevance, freshness, authority, diversity, safety, and result presentation. A result set can score high even when two acceptable pages swap positions.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

A good rubric separates correctness, completeness, grounding, tone, refusal behavior, and actionability. A fluent answer should not receive a high score if it invents policy or misses the user's real need.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

A good rubric separates functional correctness, test quality, minimality, security, maintainability, integration risk, and whether the agent changed code it should have left alone.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is evaluating 200 support answers with a rubric that scores policy correctness, completeness, tone, and next-step usefulness, while separately flagging hard failures for privacy, safety, and unsupported promises. The release report shows both score distribution and blocker count.

Expert Notes

At expert level, test the rubric itself. Measure reviewer agreement, track which dimensions cause confusion, maintain anchor examples, version rubric changes, and avoid changing the rubric mid-experiment unless you restart or clearly segment the results.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

Power analysis sounds advanced, but the everyday version is simple: a small flashlight will not reveal everything in a huge dark room. A small eval will not reveal every small quality difference.

The gentle starting question is not "Which equation do we use?" It is "What size of improvement would change our decision?" Once the team answers that, the math helps decide whether the planned sample is capable of seeing that improvement.

Overview With Examples

Power analysis asks whether your evaluation has enough data to detect the effect you care about. Minimum detectable effect asks how large a change must be before the test is likely to notice it.

For example, 30 samples might detect a huge quality drop, but it probably will not reliably detect a tiny 0.1-point improvement. That is not a failure of math. It is a mismatch between sample size and decision.

Many teams run an eval, see no statistically significant difference, and conclude that two systems are the same. That can be wrong. The test may simply be too small to detect the difference.

The first question should be product-driven: what improvement is worth acting on? A 0.05-point score improvement may not justify a more expensive model. A 0.5-point improvement with lower failure rate might.

Power analysis helps testers design the evaluation before looking at results. If the team wants to detect a 0.3-point score improvement with reasonable confidence, the sample size should be chosen for that goal.

This changes the conversation. Instead of saying, "We tested 40 examples and saw nothing," the tester can say, "With 40 examples, this eval can only detect large changes. It is underpowered for the small improvement product is asking about."

Power also matters for failure rates. Detecting a drop from 10% failures to 5% is much easier than detecting a drop from 1.0% to 0.5%. Rare events require more data or targeted tests.

A good evaluation plan states the target effect size, expected noise, sample size, and decision threshold before the run. That keeps teams from inventing the goal after seeing the results.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is deciding that a prompt change must improve average score by at least 0.3 points or reduce policy failures by at least 30% to be worth shipping. The tester then chooses a sample size large enough to detect changes at that scale.

Expert Notes

Expert teams distinguish statistical power from business value. High power helps detect a chosen effect, but the minimum meaningful effect should come from product risk, user impact, cost, and operational tradeoffs.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

Multiple comparisons are the statistics version of repeatedly rolling dice and only remembering the lucky roll. If you look at enough variants, slices, and metrics, something will appear impressive by chance.

The math can get formal, but the practical warning is plain: the more places you searched for a win, the less you should trust the one shiny win you found unless you confirm it on fresh evidence.

Overview With Examples

Multiple comparisons are a trap in AI evaluation. If you compare many prompts, many models, many categories, and many metrics, some result will look impressive by chance.

For example, testing 20 prompt variants and picking the one with the best p-value is not the same as proving that variant is truly best. You may have selected noise.

The problem is simple. Every statistical test has some chance of producing a false positive. Run one test and that risk may be acceptable. Run a hundred tests and the chance that at least one looks significant by luck can become large.

AI teams do this constantly. They compare many prompt rewrites, many temperatures, many model versions, and many category breakdowns. Then they celebrate the best-looking result without accounting for how many chances they gave luck to win.

This also happens in dashboards. One category turns green, one metric improves, one segment looks great, and the team treats it as a discovery. But if the team inspected dozens of cuts, the one shiny result may not mean much.

The fix is not to stop exploring. Exploration is useful. The fix is to label exploration as exploration and confirm important discoveries with a fresh holdout set or a planned test.

For release decisions, predefine the primary metric and key segments. Secondary metrics can provide color, but they should not silently become the main evidence after the run.

Teams can also use correction methods or false-discovery controls when many formal comparisons are unavoidable. The practical lesson is even simpler: the more you looked, the less impressed you should be by the single best thing you found.

Examples

Web Search Example

Statistics help decide whether a ranking change really improved relevance or whether a small lift came from noise in the query sample. Report the effect size and uncertainty, not just a leaderboard number.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Statistics help compare two prompts, models, or policies across many conversations. A small average-score improvement is only meaningful if the interval, severe-failure rate, and business impact support the release.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Statistics help compare agent versions across many tasks: pass rate, regression rate, review score, security flags, and time-to-fix all need uncertainty estimates before declaring one agent better.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, power analysis helps decide how many positive and negative cases are needed to detect a meaningful change in sensitivity or specificity. If the expected improvement is small, the study may need many more cases than a team expects, especially for rare findings.

Testing/Quality Example

A testing/quality example is comparing 12 candidate prompts. The team uses a development sample to narrow to two finalists, then runs those finalists once on a fresh evaluation set. The release decision uses the fresh set, not the luckiest development result.

Expert Notes

At expert level, separate exploratory analysis, confirmatory analysis, and monitoring. Use holdout sets, preregistered primary metrics, adjusted thresholds, or false-discovery-rate methods when many comparisons are part of the process.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Adversarial and red-team sampling deliberately looks for failure. It is not trying to represent average use. It is trying to expose privacy leaks, jailbreaks, unsafe advice, prompt injection, policy bypasses, and tool misuse.

For example, a normal user may ask for refund help. An adversarial user may hide malicious instructions in a document, ask the agent to ignore policy, or trick it into exposing another user's data.

Random sampling is necessary, but it is not sufficient for high-risk AI systems. If a failure is rare under normal traffic but catastrophic when triggered, random sampling may miss it.

Red-team cases should target the system's boundaries. What must it refuse? What must it never reveal? What actions require confirmation? What external content should not override trusted instructions?

For LLM systems, adversarial inputs may include jailbreak phrasing, role-play pressure, encoded instructions, malicious retrieved documents, conflicting policies, emotional manipulation, and requests that mix allowed and prohibited intent.

For agents, adversarial cases should test tool permissions, irreversible actions, payment flows, account changes, data exfiltration, and recovery from bad tool results.

The report should not blend red-team results into the average as if they were ordinary traffic. Red-team results are risk evidence. A low average score on adversarial tests may be expected; a single severe bypass may be a blocker.

A mature strategy uses both: random samples to estimate everyday quality and adversarial samples to test whether the system can be trusted under pressure.

Examples

Web Search Example

Sampling should include head queries, long-tail queries, navigational queries, fresh-news queries, ambiguous queries, local queries, multilingual queries, and adversarial or unsafe queries.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Sampling should include common support questions, confused users, angry users, multilingual users, policy boundaries, privacy-sensitive cases, tool-use cases, and rare but severe failures.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Sampling should include small bug fixes, multi-file changes, dependency updates, security-sensitive code, flaky tests, ambiguous requirements, and tasks where the correct move is to ask before editing.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, multiple comparisons are everywhere: many diseases, scanners, patient groups, image views, and thresholds. If the team checks enough slices, something will look improved by chance. Control false discoveries and treat surprising wins as hypotheses to confirm.

Testing/Quality Example

A quality example is adding a 150-case red-team suite for a support agent: prompt injection in retrieved documents, requests for private account data, refund-policy bypass attempts, unsafe escalation instructions, and tool calls that should require confirmation.

Expert Notes

Expert red-team programs track attack family, severity, exploitability, reproducibility, and mitigation status. They also refresh attacks frequently because users and attackers adapt once a system is deployed.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Dataset bias happens when the evaluation sample does not represent the real product problem. Coverage gaps are the places the test data simply does not reach.

For example, a support bot may perform well on English desktop refund questions and fail on Spanish mobile billing questions. The overall score can look fine while important users are underserved.

Every sample tells a story about the population it came from. If the sample is mostly easy, common, English-language, happy-path cases, the results describe that world. They do not describe the whole product.

Coverage gaps often hide in plain sight: languages, regions, accessibility needs, product tiers, customer segments, prompt lengths, device types, new users versus power users, and high-risk policy categories.

Bias also appears in labels. If reviewers are not trained on regional language, domain policy, or accessibility expectations, the evaluation may reward the wrong behavior.

The fix starts with a coverage map. List the important segments and risks. Decide which ones need representative sampling and which ones need targeted stress tests. Then report results by segment.

Averages should never be allowed to erase vulnerable or high-impact groups. A model that performs well overall but fails a protected, regulated, or strategically important segment is not simply "mostly good."

Dataset quality is not administrative housekeeping. It is the foundation of whether the evaluation can be trusted.

Examples

Web Search Example

Bias testing asks whether different groups, languages, regions, businesses, or viewpoints are represented fairly and whether harmful stereotypes are amplified in ranking or snippets.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Bias testing asks whether the assistant treats users consistently across identity, dialect, ability, geography, and socioeconomic context while still respecting safety and policy.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Bias testing asks whether the agent overfits to certain frameworks, coding styles, languages, platforms, or assumptions about users, accessibility, names, locations, and data.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, coverage gaps can be dangerous. If the dataset underrepresents certain ages, skin tones, body types, devices, hospitals, or rare conditions, the model may look strong overall while failing people who most need reliable detection.

Testing/Quality Example

A testing/quality example is auditing the eval set and discovering that 80% of cases are English FAQ questions, while real traffic includes billing disputes, account recovery, Spanish support, and accessibility-related requests. The tester rebuilds the sample before trusting the score.

Expert Notes

At expert level, maintain a coverage matrix with population share, risk weight, sample count, pass rate, confidence interval, and known exclusions. Make omissions explicit instead of letting them become hidden assumptions.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

AI quality is rarely a single metric. A change can improve answer quality while increasing latency, cost, token use, tool calls, or escalation rate.

For example, a larger model may raise average score from 8.0 to 8.4 but double cost and push p95 latency beyond the product target. The quality number alone does not decide the release.

Teams often talk about quality as if more is always better. In real systems, more quality may come with tradeoffs. A longer answer may be more complete but less usable. A bigger model may be safer but too slow. A retrieval-heavy workflow may be more grounded but more expensive.

This is why release reports should include cost and latency next to quality metrics. A quality gain that destroys responsiveness may harm users. A cheaper model that slightly reduces average score but dramatically lowers latency may be the right choice for low-risk cases.

Tradeoffs also vary by segment. High-risk policy answers may deserve slower, more expensive review. Low-risk creative suggestions may not.

The decision should be explicit. What are the target latency bounds? What is the budget per task? Which categories justify higher cost? Which quality failures are unacceptable no matter how cheap the system is?

A single blended score cannot answer those questions. The tester should show a multi-metric view and call out tradeoffs in plain language.

The mature pattern is routing. Use cheaper, faster paths for low-risk work and stronger paths for high-risk or ambiguous work.

Examples

Web Search Example

Quality must be weighed against latency and infrastructure cost. A ranking or summarization step that improves relevance slightly may still be wrong if it makes search slow or too expensive.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Quality must be weighed against token cost, model latency, privacy, region, reliability, and resolution value. A larger model is not automatically better if a smaller one solves the case safely.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Quality must be weighed against token cost, tool time, test runtime, review burden, security risk, and developer time saved. A costly agent is only worth it when the patch value clears the validation cost.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is comparing two models where Model B improves average score by 0.4 but increases p95 latency from 1.2 seconds to 4.8 seconds and cost by 3x. The recommendation might be to use Model B only for high-risk categories or escalation cases.

Expert Notes

Expert teams build Pareto views: quality, safety, latency, cost, and escalation rate. A release candidate is not automatically best because it wins one metric; it is best when it sits on the right frontier for the product's risk and economics.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Regression testing for non-deterministic systems is not about freezing every output. It is about detecting when behavior gets worse on the properties that matter.

For example, a summary can change wording without regressing, but it regresses if it drops a key risk, invents a fact, or becomes less usable for the target user.

Traditional regression tests often compare current output to an expected output. That is useful for deterministic systems. For LLMs, search, ranking, and agents, exact comparison can create noise.

The better pattern is invariant-based regression. Define what must remain true: required facts, policy boundaries, refusal behavior, ranking relevance, citation grounding, tool permission checks, or latency limits.

A regression suite should contain known important cases, past failures, high-risk categories, and representative examples. Each case should define the properties being protected.

Expected outputs can still be useful as examples, but they should not be treated as the only acceptable response unless the product truly requires exact wording.

Regression reports should distinguish acceptable drift from true degradation. If the wording changed but the answer stayed correct, do not burn the team's attention. If the score dropped, a hard failure appeared, or a high-risk category weakened, slow down.

Baseline refresh is part of the work. Old expected outputs can become stale when products, policies, or user needs change. Refresh deliberately, not casually.

Examples

Web Search Example

This concept applies to query understanding, ranking, retrieval, snippets, safety, latency, and result satisfaction. The test should ask how regression testing when outputs keep changing changes what users see across realistic query slices.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

This concept applies to conversation quality, grounding, tone, refusal, memory, tool use, escalation, and recovery. The test should ask how regression testing when outputs keep changing changes the user's outcome across realistic conversations.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

This concept applies to task understanding, file selection, code edits, tests, tool use, reviewability, security, and maintainability. The test should ask how regression testing when outputs keep changing changes the quality of generated patches across realistic coding tasks.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is rerunning 500 golden cases after a model upgrade and scoring each output against invariant criteria. The report flags policy regressions, missing required facts, lower category scores, and newly introduced severe failures.

Expert Notes

At expert level, keep separate baselines for examples, rubrics, labels, model versions, and judge versions. A regression can come from the product, the evaluator, the dataset, or the policy changing underneath the test.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Tool-using agents are harder to evaluate than single-turn answers because they act. They choose steps, call tools, interpret results, and may create side effects.

For example, a travel agent might search flights, compare policies, ask for confirmation, book a ticket, and send an email. Quality includes every step, not just the final message.

Agent testing should evaluate the plan, the tool calls, the arguments passed to tools, the interpretation of tool results, the handling of errors, and the final user-facing response.

The most important questions are often about permission and control. Did the agent ask before taking an irreversible action? Did it expose data it should not? Did it continue when a tool returned ambiguous or contradictory information?

Multi-step workflows also create compounding errors. A small misunderstanding early in the flow can lead to a bad tool call, which leads to a misleading final answer.

Test cases should include happy paths, missing information, tool failures, conflicting data, malicious tool output, permission boundaries, and recovery paths.

Metrics should go beyond answer quality. Track task completion, tool-call correctness, unnecessary tool calls, unsafe attempted actions, confirmation compliance, recovery success, and user-visible explanation quality.

Agents should be judged on whether they achieved the user's legitimate goal safely, not whether they sounded confident while doing something risky.

Examples

Web Search Example

Agentic behavior appears when the system rewrites queries, chooses retrieval tools, summarizes results, or takes follow-up actions. Prefer bounded steps when the search flow is known.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Agentic behavior appears when the assistant plans, calls tools, remembers context, retries, or escalates. Score the path it took, not only the final message.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Agentic behavior is the product: reading files, forming a plan, editing code, running tests, recovering from errors, and deciding when to stop. Score the trajectory, not just the patch.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is testing an account-management agent with cases for password reset, plan downgrade, refund request, suspicious login, and account deletion. The tester checks whether the agent uses the right tools, asks for confirmation, respects policy, and recovers from tool errors.

Expert Notes

Expert agent evals use traces as first-class artifacts. Score the final answer and the trajectory: plan quality, tool selection, arguments, observations, state updates, side effects, and escalation decisions.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Human review workflows define how evaluation evidence turns into action. Some cases can be handled by deterministic checks or LLM judges. Others need expert review, policy review, security review, legal review, or product escalation.

For example, a low-risk style issue can stay automated, but a privacy leak, medical-risk answer, or account-deletion action should have a clear human escalation path.

A good evaluation system does not pretend every decision can be automated. It defines routing rules. Which outputs are auto-accepted? Which are sampled for audit? Which are always escalated?

Escalation rules should be based on risk, confidence, disagreement, severity, and reversibility. A low-confidence judge score in a high-risk category should not be treated like a low-confidence score on a harmless creative prompt.

Human review also needs workflow design. Reviewers need context, rubric definitions, source documents, prior decisions, and a way to label failure reasons consistently.

Feedback loops matter. If human reviewers repeatedly overturn the judge in one category, the judge prompt, rubric, or product behavior needs attention.

Escalation should be visible in release reports. A system that requires human review for 40% of cases may be safe but operationally expensive. That is still quality evidence.

The goal is a reliable human-AI evaluation process, not a fantasy of full automation.

Examples

Web Search Example

Agentic behavior appears when the system rewrites queries, chooses retrieval tools, summarizes results, or takes follow-up actions. Prefer bounded steps when the search flow is known.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Agentic behavior appears when the assistant plans, calls tools, remembers context, retries, or escalates. Score the path it took, not only the final message.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Agentic behavior is the product: reading files, forming a plan, editing code, running tests, recovering from errors, and deciding when to stop. Score the trajectory, not just the patch.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, human review is not a fallback detail. Define which cases require radiologist review, second-reader workflows, uncertainty escalation, audit sampling, and disagreement resolution. The AI should support clinicians, not silently replace clinical judgment where evidence is thin.

Humanoid Robot Example

For humanoid robots and embodied AI, escalation means more than asking a person a question. It can mean freezing motion, lowering force, backing away, requesting supervision, or entering a safe pose. Human review workflows must account for physical state and response time.

Testing/Quality Example

A quality example is routing all critical safety failures, privacy flags, legal-policy conflicts, low-confidence high-risk judge decisions, and judge-human disagreements into human review before release approval.

Expert Notes

At expert level, track reviewer queue time, overturn rate, escalation precision, escalation recall, reviewer agreement, and category-level escalation load. A review workflow is itself a system that needs quality metrics.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Eval data management is the discipline of keeping evaluation artifacts traceable. It sounds boring until a team cannot explain why last month's score and this month's score are different.

For example, a quality score can change because the model improved, the judge changed, the rubric changed, the sample changed, or labels were updated. Without versioning, those causes blur together.

Non-deterministic evaluation produces many artifacts: prompts, model versions, retrieval snapshots, datasets, labels, rubrics, judge prompts, judge models, scoring code, random seeds, traces, outputs, and reports.

Each artifact should have an identity. When a result is reported, the team should know exactly which versions produced it.

This matters for comparisons. If Version B used a different judge prompt than Version A, the comparison may not be fair. If the dataset changed, the trend line may be measuring sample drift instead of product quality.

Good eval data management also protects institutional memory. Production failures become golden cases. Rubric changes explain why older scores are not directly comparable. Label updates show how the team's definition of quality evolved.

Privacy and access controls belong here too. Eval datasets often contain real user examples or sensitive business policy. The team should know what can be stored, who can see it, and how long it is retained.

A credible evaluation is not just a score. It is a score with provenance.

Examples

Web Search Example

An eval suite should include realistic queries with expected relevant documents or graded relevance labels, plus benchmark-style checks for ranking quality such as NDCG or recall at k.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

An eval suite should include realistic conversations with expected behaviors, rubric scores, safety checks, grounding checks, and examples where the right answer is to ask, refuse, or escalate.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An eval suite should include runnable tasks with repos, failing tests, hidden regressions, security checks, code-review rubrics, and cases where no code change should be made.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is a release report that links to dataset version, model version, system prompt version, judge prompt version, rubric version, scoring script hash, run timestamp, and sampled output artifacts.

Expert Notes

Expert teams treat evals like experiments and production telemetry at the same time. They keep immutable run records, separate raw data from derived labels, document schema changes, and make comparisons only between compatible runs.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Gentle Math Introduction

NDCG looks mathematical because it has a formula, but the intuition is friendly: good search results should put the most useful items near the top, where users actually look.

The metric gives more credit for relevant results at high ranks than low ranks. Before memorizing the calculation, remember the product idea: rank order matters, and a great result buried on page two is not as valuable as a great result at the top.

Overview With Examples

NDCG, or normalized discounted cumulative gain, is a metric for ranked results. It rewards relevant results near the top of the list more than relevant results buried lower down.

For example, a search engine that puts the best answer first should score better than one that puts the same answer on page two, even if both technically returned it.

Search and recommendation quality is not just about whether a relevant item appears somewhere. Rank matters. Users often inspect the top few results and stop.

NDCG starts with relevance judgments. A result might be judged 0 for irrelevant, 1 for somewhat relevant, 2 for relevant, and 3 for highly relevant. The metric gives more credit when high-relevance items appear earlier.

The discount matters because position matters. A highly relevant result at rank 1 is much more valuable than the same result at rank 20. NDCG captures that intuition.

The normalized part compares the actual ranking against the ideal ranking for that query. A score near 1.0 means the ranking is close to ideal. A lower score means relevant results are missing or poorly ordered.

NDCG is useful for comparing search algorithms, retrieval systems, recommendation lists, RAG retrieval quality, and ranked support suggestions.

It is not the only metric. You may also need recall, precision, click behavior, no-result rate, latency, diversity, safety filters, and business constraints. But NDCG is one of the most practical relevance metrics for ranked lists.

Examples

Web Search Example

NDCG is directly useful because it rewards putting highly relevant results near the top and discounts relevant results that appear too low to matter.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

The same ranked-quality idea appears when the bot chooses sources, suggestions, next actions, or tool results. The most useful and safest option should appear first.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Ranked relevance appears when the agent chooses files, errors, candidate fixes, or tests to run. The most likely and highest-value evidence should be surfaced first.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is judging the top 10 search results for 500 queries on a 0-3 relevance scale, then comparing NDCG@10 for the old and new ranking models. The report also breaks NDCG down by query type, language, and high-risk product area.

Expert Notes

At expert level, choose the cutoff deliberately, such as NDCG@5 or NDCG@10, based on how many results users actually inspect. Watch for label quality, position bias in click data, query mix drift, and improvements that help common queries while hurting rare critical queries.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Non-deterministic systems produce noisy measurements. If you keep rerunning the same evaluation and report only the best result, you are not measuring quality. You are selecting a lucky high-water mark.

For example, a prompt may average 8.0 across repeated runs but occasionally score 8.6 by chance. Reporting the 8.6 as the truth is wrong and will mislead the team.

This failure mode is common because it feels productive. The team reruns the eval, tweaks a prompt, reruns again, changes a judge instruction, reruns again, and eventually sees a new best score. Everyone wants to believe the high score is progress.

Sometimes it is progress. Often it is variance. Non-deterministic systems, sampled datasets, LLM judges, and small evaluation sets all create noise. The maximum observed result across many tries is biased upward.

High-water marks are especially dangerous when the team does not log every run. If only the best run survives, the evidence trail disappears. The team forgets how many attempts failed to reproduce the win.

The fix is to report all runs, not just the best run. Show the mean across runs, the spread, the confidence interval, and whether the improvement reproduces on a fresh holdout set.

A new high score should be treated as a lead, not proof. It earns a confirmation run. It does not earn a release by itself.

The same rule applies to cherry-picked examples. A stunning generated answer shows what the system can do. It does not show how often the system does it.

Examples

Web Search Example

A lucky offline run can create a new high-water mark even when the ranking change is not reliably better. Treat the best run as a clue, not proof.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

One excellent response or one strong eval run can be pure variance. Do not promote a prompt because it produced a beautiful answer once.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

One impressive demo patch can hide a weak agent. Promote only when repeated tasks show reliable correctness, useful tests, and low regression risk.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is running a prompt eval ten times and seeing scores from 7.7 to 8.5. The tester reports the distribution and requires a fresh confirmation sample before claiming improvement, instead of celebrating the 8.5 as the new baseline.

Expert Notes

At expert level, treat repeated evaluation as a multiple-comparisons problem. Track every run, predefine stopping rules, preserve holdout sets, and estimate performance from the full run distribution rather than the maximum observed score.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Evals are structured ways to measure model or system behavior. They can be public benchmarks, private product evals, red-team suites, human preference studies, or continuous production monitors.

For example, MMLU measures broad academic knowledge, HumanEval measures code-generation tasks, SWE-bench measures issue-resolution on real software repositories, GPQA measures hard science questions, Chatbot Arena measures human preference, and HELM tries to compare models across many scenarios and metrics.

Popular evals are useful because they give teams a shared language. They let people compare systems, spot broad capability changes, and notice when a model is obviously behind the frontier.

But benchmarks are not product truth. A model can score well on MMLU and still fail your refund policy. It can perform well on HumanEval and still produce unsafe code in your stack. It can win preference battles and still be wrong in high-risk domains.

Computer-use benchmarks are especially tricky. WebArena, OSWorld, WorkArena, browser-use tasks, and screen-based agent benchmarks try to measure whether agents can operate software. These are valuable, but often poorly defined. The environment may be brittle, success criteria may be ambiguous, and the official answer can be incomplete, stale, or simply wrong.

Many benchmark tasks also hide huge variance. A web task can fail because a page changed, a selector moved, an account state differed, a modal appeared, or the benchmark expected one path when another path also completed the task. Treating that as a clean model failure is sloppy.

Some eval datasets contain wrong answers. Some contain outdated facts. Some reward test-taking tricks rather than practical competence. Some are contaminated because training data included the benchmark or close variants. Some compress a complex workflow into a single pass/fail answer that loses important quality information.

That does not mean public evals are useless. It means testers should read evals like testers. Ask what the eval actually measures, how labels were created, how failures are judged, whether the task still reflects reality, and whether the metric matches the product decision.

The best strategy is layered. Use public benchmarks for broad signals. Use domain evals for product-specific quality. Use adversarial suites for known risks. Use live sampling for current reality. Use monitoring to detect drift after release.

Examples

Web Search Example

An eval suite should include realistic queries with expected relevant documents or graded relevance labels, plus benchmark-style checks for ranking quality such as NDCG or recall at k.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

An eval suite should include realistic conversations with expected behaviors, rubric scores, safety checks, grounding checks, and examples where the right answer is to ask, refuse, or escalate.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An eval suite should include runnable tasks with repos, failing tests, hidden regressions, security checks, code-review rubrics, and cases where no code change should be made.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

High-Stakes Extensions

Medical Imaging / Detection Example

For medical imaging and detection, public benchmarks can be useful but insufficient. A benchmark may not match the target hospital, scanner mix, prevalence, labeling protocol, or clinical workflow. Treat benchmark performance as entry evidence, then validate locally before deployment.

Testing/Quality Example

A testing/quality example is evaluating an AI browser agent on a public web benchmark, then manually reviewing a sample of failures. The team may discover that some tasks changed, some expected answers are stale, and some agent paths succeeded differently from the benchmark oracle. The report separates true model failures from benchmark noise.

Expert Notes

At expert level, audit benchmarks before trusting them. Track task validity, label quality, contamination risk, environment drift, oracle ambiguity, metric fit, and inter-rater agreement. A leaderboard score is an input, not a release decision.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

A recent study asked whether language models can pass software testing certification exams using 30 ISTQB sample exams across foundation, advanced, specialist, and expert categories. The result was blunt: two models passed all 30 sample certification exams by scoring at least 65%.

That does not mean an AI officially became ISTQB certified. It means certification-style exam questions are increasingly solvable by models, which should make testers rethink what professional competence really means.

This matters because certification exams often reward terminology, syllabus recall, and exam-pattern reasoning. Those skills are not worthless, but they are no longer enough to define testing expertise in an AI world.

If a model can answer many certification questions, then the valuable human work moves up the stack. The tester must define the right risk, build the right rubric, choose the right sample, interpret uncertainty, challenge the benchmark, and explain the release decision.

The ISTQB result should not be read as "certifications are useless." A shared vocabulary can help teams communicate. A syllabus can introduce important concepts. But passing a knowledge exam is different from designing a credible evaluation for a live AI product.

There is another lesson: exams are evals too. They have oracles, wording assumptions, possible ambiguous answers, syllabus boundaries, and pass thresholds. If AI can pass them, testers should ask what the exam is measuring and what it is not measuring.

The same applies to internal training tests. If the assessment only checks recall, AI will do well. If it asks someone to investigate a flaky non-deterministic failure, build a sampling plan, calibrate an LLM judge, or defend a release recommendation, the assessment becomes more meaningful.

The future tester does not win by knowing definitions that a model can retrieve. They win by turning messy product risk into measurable evidence and responsible action.

Examples

Web Search Example

An eval suite should include realistic queries with expected relevant documents or graded relevance labels, plus benchmark-style checks for ranking quality such as NDCG or recall at k.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

An eval suite should include realistic conversations with expected behaviors, rubric scores, safety checks, grounding checks, and examples where the right answer is to ask, refuse, or escalate.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

An eval suite should include runnable tasks with repos, failing tests, hidden regressions, security checks, code-review rubrics, and cases where no code change should be made.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A quality example is using AI to answer certification-style testing questions, then asking a human tester to critique a flawed eval report. The AI may pass the quiz, but the stronger test of professional skill is whether the human can spot bad sampling, wrong oracles, missing risk categories, and overclaimed confidence.

Expert Notes

At expert level, treat certification performance as a benchmark with limits. The cited ISTQB study used sample exams, not official proctored certification records. The result is still important because it shows that exam-style testing knowledge is increasingly automatable, while real evaluation design remains context-heavy.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

The testing bias material starts with a blunt premise: you cannot eliminate all bias. The useful goal is to find bias, understand it, and decide whether it is acceptable, harmful, or intentionally useful.

For example, a search crawler that starts from popular sites will overrepresent well-linked, well-formed, commercial, and English-language pages. That bias may improve mainstream results while making the system worse for obscure, local, underfunded, or poorly connected sources.

Bias begins with data sourcing. If your dataset only contains public pages, you have excluded private knowledge, paywalled content, internal tools, and communities that do not publish in the same way. If your production traffic comes from one default channel, it may represent that channel's users more than your true market.

Sampling adds more bias. A sample taken from one hour, one region, one machine, one week, or one season can teach the system a distorted version of reality. A search system trained on weekday office traffic may behave differently from one trained on weekend home traffic.

Production data is tempting, but it can create feedback loops. Click data in search is a classic example: users tend to click higher-ranked results partly because they are higher-ranked, not only because they are better. Training on those clicks can reinforce the old ranking bias.

Sample size also affects bias. Small samples can miss minority groups or overrepresent them by accident. The tester needs to understand the texture of the data: which segments exist, how frequent they are, how noisy they are, and which ones matter even if they are rare.

Train/test splits are part of the bias story too. A bad test set gives the wrong students A grades. If the test data mirrors the training data too closely, the evaluation may reward memorization. If the test set misses important segments, the model can look good while failing real users.

The bias tester's job is not to demand impossible purity. It is to document what the dataset sees, what it cannot see, what it overweights, what it underweights, and how those choices will show up in the product.

Examples

Web Search Example

Bias testing asks whether different groups, languages, regions, businesses, or viewpoints are represented fairly and whether harmful stereotypes are amplified in ranking or snippets.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Bias testing asks whether the assistant treats users consistently across identity, dialect, ability, geography, and socioeconomic context while still respecting safety and policy.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Bias testing asks whether the agent overfits to certain frameworks, coding styles, languages, platforms, or assumptions about users, accessibility, names, locations, and data.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is auditing a search training sample by source, language, geography, device, time of day, day of week, query frequency, and user segment. The report identifies which groups are overrepresented, which are missing, and which biases are acceptable business choices versus quality risks.

Expert Notes

At expert level, bias testing treats the dataset as a product surface. Track provenance, sampling windows, exclusion rules, coverage gaps, leakage between train and test sets, and feedback loops from production behavior. Every data-selection rule is also a product decision.

Major Concepts

Ask AI About This Chapter

Open a focused conversation about this chapter.

Overview With Examples

Labeling bias appears in two places: the labeling process and the raters themselves. Both shape what the model later treats as truth.

For example, a search rater guideline that rewards authority may favor professors, governments, and large institutions over firsthand experience, smaller sites, or communities with less formal status.

Labeling guidelines are not neutral. They encode values. A rule against distracting ads may improve user experience but also favor organizations that can afford ad-free publishing. A rule favoring authority may fight misinformation but also suppress useful lived experience.

Rater disagreement is not noise to sweep away. Ambiguous queries produce real disagreement because users have different intent. The query "bush" might mean a plant, a president, a band, or something else depending on the person and moment.

That disagreement is entropy in the training data. If you average it away too early, the model learns a flattened version of user intent. If you ignore it, you may miss minority interpretations that matter.

Overlap is one mitigation. Ask multiple raters to label the same item, then measure where they agree and where they diverge. High-variance items need more inspection. Low-variance items can still need overlap when ranking second, third, and fourth best results matters.

Cleaning labels can also create bias. Removing misspellings, rare strings, outlier raters, strange examples, or unpopular answers may make metrics cleaner while making the model less useful for real users.

The dangerous pattern is invisible cleanup. If a vendor silently removes raters who disagree with peers, the dataset may become more consistent but less representative.

Examples

Web Search Example

Raters can judge query-result relevance, freshness, spam, trustworthiness, and whether the result satisfies the likely intent. Disagreement often reveals ambiguous intent or weak guidelines.

In a test suite, make this concrete with query slices, expected relevant evidence, unacceptable outcomes, and ranking metrics. Include common queries, edge cases, negative cases, security-sensitive cases, and high-value business queries so the result is not just a polished demo.

Chatbot Example

Raters can judge correctness, helpfulness, policy compliance, empathy, grounding, and escalation quality. Disagreement often reveals vague rubrics or missing examples.

For the chatbot version, evaluate full conversations. Capture the user's intent, the required source facts, the policy boundary, the expected tone, and the point where the assistant should answer, ask a clarifying question, refuse, or escalate.

AI Coding Agent Example

Reviewers can judge whether a patch is correct, minimal, idiomatic, secure, tested, and easy to maintain. Disagreement often reveals unclear engineering standards or hidden product intent.

For the coding-agent version, evaluate the whole trajectory: task interpretation, files inspected, commands run, code changed, tests attempted, reviewability, security, and whether the agent stopped at the right time. A good case makes both the expected patch and the forbidden side effects visible.

Testing/Quality Example

A testing/quality example is running a rater-overlap study on high-impact labels. The tester tracks agreement, disagreement reasons, demographic skew, guideline sensitivity, and which labels change when instructions are rewritten. The output is a bias report, not just a label-quality score.

Expert Notes

At expert level, labeling tests should separate harmful inconsistency from meaningful plurality. Use agreement metrics, entropy analysis, rater demographics, guideline A/B tests, and adjudication logs. Do not let the cleanup process erase the very users the system needs to serve.

Major Concepts