--- name: tai-ch050-using-testfoo-and-promptfoo description: 'Apply chapter 50 of Testing AI, Using TestFoo and Promptfoo, as a workflow for evaluating AI and non-deterministic systems. Use for test planning, eval design, quality review, release evidence, examples, or coaching related to using testfoo and promptfoo.' --- # Using TestFoo and Promptfoo Skill name: `tai-ch050-using-testfoo-and-promptfoo` Based on **Testing AI: Engineering Confidence in AI Systems** by **Jason Arbon**. ## Purpose A lightweight eval tool turns prompt checks from vibes into repeatable tests that can run locally, in CI, and before release. ## Use This Workflow - Identify the AI behavior or release decision being evaluated. - Define realistic cases, slices, unacceptable outcomes, and evidence needed for confidence. - Choose measurements that match the risk: rubric scores, samples, intervals, traces, human review, deterministic checks, or production monitors. - Report uncertainty, severe failures, and decision impact instead of only a pass/fail result. ## Key Guidance When teams say they want TestFoo, they often mean a Promptfoo-style workflow: define prompts, providers, test cases, assertions, and scoring rules in a repeatable eval file, then run it every time the prompt, model, retrieval layer, or policy changes. For example, a support chatbot team can compare two prompts across OpenAI, Anthropic, Gemini, and a local model, run 200 policy cases, score outputs with assertions or an LLM judge, and fail the build if the pass rate drops below the release threshold. ## Apply The Approach Create representative cases, score them with explicit criteria, review severe failures separately, report uncertainty, and connect the evidence to a concrete decision. ## Expert Notes At expert level, treat TestFoo or Promptfoo as eval infrastructure, not a substitute for evaluation design. Version configs, lock datasets, track judge model changes, separate exploratory runs from release gates, and periodically compare automated scores against human raters.