The 3.2% shows up in the audit report first —3,200 malformed JSON payloads per 100,000 API calls, each a silent business logic failure.
One model string change caused it. No dashboard caught it. HTTP stays green while structured outputs silently degrade, and you trace the wreckage back three sprints.
CI-integrated eval catches semantic regressions at the pull request —same force as unit tests blocking null pointers.
This pipeline spans deterministic checks through LLM-as-Judge scoring, with a closed loop that turns today’s failures into tomorrow’s CI test cases.
Why “It Looks Right” Scales to About 50 Reviews —Then Fails
Why “Does It Look Right?” Doesn’t Scale
A human reviewer staring at LLM outputs is unreliable in three specific, measurable ways:
- Fatigue. Accuracy drops sharply after ~50 consecutive reviews. Your 51st judgment is meaningfully worse than your 5th —and you won’t notice the decline.
- Inconsistency. Inter-rater reliability between two human reviewers on the same set of LLM outputs typically falls below 0.6 —meaning two qualified people disagree on “is this correct?” 40% of the time.
- Cost and latency. 1,000 outputs at 90 seconds per review = 25 hours of human time. That’s $750-2,500 per eval run —and it takes days to schedule.
Machine evaluation is consistent, instantaneous, and nearly free. But “machine evaluation” isn’t one thing —it’s a stack of three primitives you combine based on what you’re testing.
The Three Evaluation Primitives
Layer 1: Deterministic Checks. Microseconds. Zero API cost. JSON Schema validation. Exact string match. Tool-call success/failure counting. Citation existence verification. Refusal regex matching. These catch roughly 50% of real-world failures —and they’re the only layer that can run on every single request without budget concerns. Start here.
Layer 2: Embedding-Based Metrics. Milliseconds. ~$0.001 per check. BERTScore. Cosine similarity to a reference answer. Useful when you have “gold” answers and need paraphrase-tolerant similarity judgment. Not useful for open-ended generation where multiple valid answers exist.
Layer 3: LLM-as-Judge. Hundreds of milliseconds. $0.01-0.10 per check. A capable LLM scores outputs against rubrics using chain-of-thought reasoning. Catches semantic failures that deterministic checks miss —unfaithful summarization, unhelpful responses, incorrect refusal. Requires calibration against human labels and active bias mitigation (position bias, verbosity bias, self-enhancement bias).
The 6-Layer Production Evaluation Stack
Dataset —Metrics/Rubrics —Judge —CI Gate —Production Observation —Closed LoopBreak any link and evaluation becomes an offline report nobody reads. The dataset samples from production, weighted toward failures. The rubrics define “correct” in measurable terms. The judge scores consistently and at scale. The CI gate blocks regressions before merge. Production observation catches what CI misses. The closed loop turns today’s production failure into tomorrow’s CI test case. Six layers. One pipeline. No gaps.
Why Systematic Testing Changes Everything
The Hidden Cost of Regression
3.2% silently malformed JSON responses means 3,200 broken outputs per 100,000 API calls. If those outputs drive downstream actions, you have 3,200 business logic failures —and you won’t discover them until the financial reconciliation doesn’t match, which could be weeks.
An eval pipeline catches the 3.2% in CI, before the model string change reaches production. The cost of setting up the pipeline is less than the cost of one incident.
Prompt Drift Detection
Minor model version bumps —gpt-5.5-2026-07-01 to gpt-5.5-2026-07-15 —don’t come with behavior change notes. The changelog says “improved instruction following.” Your structured extraction accuracy dropped 4 points. You won’t know unless you’re running the same eval suite against every model version.
The Multi-Provider Evaluation Problem
If you route between models —and you should, for cost and reliability —you need eval results for every model in your routing pool. Not just the primary. A unified API endpoint lets you run the same eval suite against all models through one integration. Same request format. Same test cases. Same judge model. The only variable that changes is model: "...".
Before building your eval suite, understand the performance and cost profiles of each model in your routing pool. Our cross-provider benchmark comparison provides per-model data to ground your testing priorities.
How to Build Your Evaluation Pipeline
Step 1: Build Your Eval Dataset
The dataset is the hardest step and the one most teams underinvest in. A bad dataset gives you accurate scores on the wrong things. A good dataset is sampled from production, weighted toward failure, and refreshed weekly.
The build process:
- Randomly sample 200-500 requests from production logs. Not from your test environment. Real users ask questions test authors never think of.
- Manually annotate each: what was the correct answer? What would make the response wrong? What edge cases should the model handle?
- Stratify by difficulty: one-third easy, one-third medium, one-third hard. If your eval set is all easy queries, your scores will be inflated.
- Refresh weekly: sample new data from the last 7 days of production logs. Replace the oldest 20% of your eval set. This keeps your eval aligned with what users are actually asking —which drifts over time.
The rule that prevents the most common mistake: at least 20% of your eval set must come from historical failures. If your eval set contains only happy-path queries, you’re testing whether the model works in ideal conditions —not whether it fails gracefully in the conditions that actually occur in production.
Step 2: Define Rubrics, Not Just “Is It Good?”
Four well-calibrated rubrics beat 15 noisy ones. Each rubric needs:
- A precise definition: “faithfulness = every factual claim in the response is supported by the retrieved context”
- A scoring scale: 1-5 or 0-1
- A pass threshold: “faithfulness must be —0.85”
- Two to three scored examples as calibration references for your judge model
The core rubric set for most LLM API applications: faithfulness (are the facts correct?), answer relevance (does the response address the query?), context precision (are the retrieved chunks actually relevant?), task completion (did the model do what was asked?), refusal correctness (did the model refuse when it should have —and not when it shouldn’t have?).
Step 3: Choose and Calibrate Your Judge
GPT-4 is the most widely used LLM judge —its agreement with human annotators exceeds 80% on MT-Bench and 85%+ on G-Eval. But an uncalibrated judge gives you numbers that look precise and are systematically wrong.
The calibration process: take 50 human-annotated examples. Run them through your judge model. Calculate the correlation between judge scores and human scores. If the correlation is below 0.75 for any rubric, that rubric’s judge prompt needs work —or you need a different judge model.
Three biases that must be mitigated:
- Position bias. The judge prefers whichever response appears first. Fix: randomize response order per evaluation.
- Verbosity bias. The judge scores longer responses higher regardless of quality. Fix: score relevance and completeness as separate dimensions.
- Self-enhancement bias. The judge inflates scores for outputs generated by the same model family. Fix: use a different model family as judge than you use in production. If production runs on Claude, judge with GPT-4. Or use a dedicated judge model. For API key isolation and access controls that keep your judge and production models securely separated, see the security best practices guide.
The most overlooked rule: pin your judge model version. When you upgrade from GPT-4 to GPT-4o as judge, every historical score becomes incomparable. You can’t tell if your production model improved or your judge got stricter. Either keep the judge version constant, or recalibrate against your 50 human-annotated examples after every judge upgrade and establish a score mapping between old and new judges.
Step 4: Set Up CI Gating
Two tools, two philosophies:
- Promptfoo. Open source. Git-integrated. Declarative config in YAML. Best for teams that want eval as code, living alongside their prompts in the same repo.
- DeepEval. Python-native. 30+ built-in metrics. Decorator-based CI/CD integration. Best for teams that want eval deeply integrated into their Python test suite.
The CI gate logic, regardless of tool:
tests:
- path: eval_dataset.jsonl
asserts:
- type: python
value: |
def check_regression(output, context):
baseline = context['baseline_scores']
current = compute_scores(output)
for rubric, score in current.items():
if baseline[rubric] - score > 2:
return False, f"{rubric} dropped {baseline[rubric] - score:.1f} points"
return True, "All rubrics within threshold"Any rubric dropping more than 2 points from baseline —CI fails —merge blocked. This is not optional. A prompt change that degrades quality by 3 points on a critical rubric should never reach production. Your unit tests block a null pointer exception. Your eval gates should block a semantic regression with the same force. The prompt optimization workflow that feeds into CI gating is covered in our prompt engineering production guide.
Step 5: Production Observation + Closed Loop
CI evaluation catches pre-deployment regressions. It doesn’t catch distribution shifts, adversarial inputs, or edge cases your eval set doesn’t cover. For those, you need continuous evaluation on production traces.
Attach eval scores to your OTel spans. Any rubric sustaining a 2-5 point drop over a rolling window triggers an alert. Judge calibration methodology and the prompt optimization workflow that feeds into CI gating are both covered in the earlier steps of this guide. The chain-of-thought evaluation methodology referenced here was established by the G-Eval paper (Liu et al., 2023). Failed traces are automatically clustered by error type, prompt version, and model. Named issues with representative failures are promoted back into your offline eval dataset —manually annotated this time, with the specific failure pattern documented.
This closed loop is the difference between an eval score that trends down over time and one that keeps improving. Without it, your eval set grows stale, your model drifts, and your CI gate becomes a formality that passes while production quality degrades.
Testing Patterns for Specific API Scenarios
Testing Structured Output
JSON Schema validation alone catches ~70% of structured output failures. Add business rule checks for the remaining 30%: “price cannot be negative,” “email must match regex,” “total must equal subtotal plus tax.” Cross-field consistency checks: “if payment_method is ‘credit_card’, last_four must not be null.”
The provider-specific trap: OpenAI returns function.arguments as a JSON string. Anthropic returns tool_use.input as a JSON object. Your parser must handle both —and your eval suite must test both. Mock both response formats in CI.
For the complete request and response format reference across providers—including structured output, tool calls, and streaming—cee the chat completions API documentation.
Testing Tool Calling
Four checks, ordered by frequency of failure: (1) Tool name matches schema. (2) Required arguments are present and correctly typed. (3) Parallel tool calls don’t interfere with each other. (4) After a tool error, the model retries with corrected arguments —or escalates —not loops.
Set a hard loop limit in testing: same tool called more than three times consecutively —test fails. The model should recognize the pattern, not repeat it.
Testing RAG Quality
Three dimensions: context relevance (are the retrieved chunks about what the user asked?), answer faithfulness (is every claim in the answer supported by a retrieved chunk?), citation accuracy (does each citation point to a chunk that actually contains the cited information?). Minimum threshold: top-1 citation accuracy —90%. Below that, users will lose trust —fast.
Why Most LLM Eval Pipelines Fail Within 3 Months
Testing Only Happy Paths
Your eval set contains “What’s the return policy?” and “How do I reset my password?” —clean, friendly, well-formed queries. Production contains “i cant log in wtf???” and “URGENT: my refund still hasn’t processed and it’s been TWO WEEKS” —with typos, anger, and missing context. If your eval set doesn’t include production-like edge cases, your 95% eval score means nothing.
Fix: Sample from production logs. Ensure at least 20% of your eval set comes from historical failures —queries that previously produced wrong answers, refusals, or hallucinations.
Not Pinning the Judge Model Version
You upgraded your judge from GPT-4 to GPT-4o. All your eval scores shifted up by 3 points. Your team celebrated the “quality improvement.” Nothing changed in production. The judge just got more lenient.
Fix: Lock the judge model version. If you must upgrade, recalibrate against your 50 human-annotated examples and establish a score mapping. Without it, your eval score history is noise.
Optimizing on the Eval Set
You tuned your prompt against your eval set. 95% accuracy. Shipped. Production accuracy: 68%. The eval set contained the patterns you optimized for. Production contains everything else.
Fix: Train/eval/test split (70/15/15). The optimizer sees the training set. CI runs against the eval set. The test set is held out —you run it once before release, and the score it reports is the closest estimate of production performance you’ll get before deploying.
Once your eval gates pass in CI, deployment patterns like canary rollouts and fallback chains protect against distribution shifts in production. See the production optimization guide for rollout strategies and model failover configuration.
Further reading. Build out the structured output side of your test suite with our JSON mode and structured output comparison, which covers provider-specific format handling that your CI pipeline needs to validate. For prompt optimization methodology, see Step 4; for rollout strategies and model failover configuration, see the production optimization guide above.
FAQ
How many test cases do I need to start?
Fifty is the minimum for directional feedback —enough to tell you if a change made things better or worse, but too noisy for CI gating. One hundred to 500 cases gives you statistical significance —a single edge-case failure can’t swing your scores by more than a few points. Start at 50. Add 20-30 new cases weekly from production logs. If you’re new to programmatic LLM testing and want the API fundamentals first, our beginner’s guide to LLM APIs covers request structure before you build an eval suite.
LLM-as-Judge vs. human evaluation —how big is the gap?
GPT-4 judge agrees with human annotators >80% of the time on factual and instruction-following tasks. The gap is largest on subjective dimensions (creativity, stylistic quality). For objective dimensions —faithfulness, format compliance, task completion —LLM judges match or exceed individual human annotators, because they eliminate fatigue and inconsistency.
Should I use the same model as judge that I use in production?
No. Self-enhancement bias is real and measurable —models inflate scores for outputs from the same model family by 5-10%. Use a different model family or a dedicated judge model. If your production stack runs Claude, evaluate with GPT-4.
How do I evaluate streaming responses?
Evaluate the complete concatenated response, not individual tokens. Add performance assertions: TTFT (time-to-first-token) below your target threshold, inter-token latency P95 below budget. Streaming-specific bugs —like unescaped newlines in SSE event data breaking the parser —need dedicated deterministic checks.
How do I run the same eval suite across three providers without writing three adapters?
Send identical test cases through one base URL, changing only the model parameter between gpt-5.5, claude-sonnet-4-20250514, and gemini-3.1-pro. One request format. One eval pipeline. The only variable that changes is which model processes the prompt —which is exactly what your eval suite is designed to measure. For your first unified eval request through a single base URL, follow the quickstart guide.
LLM evaluation isn’t a phase you complete. It’s infrastructure you maintain. The 6-layer stack —dataset, rubrics, judge, CI gate, production observation, closed loop —is the minimum viable system for knowing whether your LLM-powered features are getting better or worse.
Start with 50 test cases and deterministic checks. Run them in CI this week. Add 20 cases a week from production logs. In a month, you’ll have a statistically meaningful eval set and a CI gate that catches regressions before users do.
Your eval suite shouldn’t need a provider-specific adapter for every model you test. Start testing on TokSpan —run the same 50 test cases against GPT, Claude, and Gemini through a single integration point.