Measuring What Matters

What an eval is

An eval is not a benchmark. Benchmarks measure a model's general capabilities against a standardized dataset — MMLU, HumanEval, GSM8K. Those numbers tell you whether a model is capable of doing things; they do not tell you whether your system does the thing you built it to do. That is what an eval is for.

When you build a support triage agent, the relevant question is not "what is this model's score on a reading comprehension benchmark?" The question is: "does my agent correctly classify incoming support tickets into the right queues, and does it do so reliably enough across the distribution of real tickets my customers actually submit?"

That distinction matters because model capability and task performance diverge in ways that are predictable in hindsight and invisible without measurement. A highly capable model may still fail your task due to a bad system prompt, an underspecified output format, domain vocabulary it was not trained on, or edge cases your prompt does not handle. An eval makes those failures visible before they reach production.

What an eval is not

An eval is not a one-time check you run before launch. It is a regression suite that runs on every change — to the model, to the prompt, to the tools, to the retrieval logic. The value accumulates over time as you add cases that represent failures you have seen.

An eval is not a vibe check. "It seems to work on the examples I tried" is not an eval. You need a repeatable, automated process that produces a score you can track over time and compare against a threshold.

An eval is not a test of the model's general intelligence. You are testing your system — the model plus the prompt plus the tools plus the retrieval logic — against your specific task. If the system passes your eval but fails in production, your eval has a coverage gap.

Anatomy of an eval

Every eval has the same four components: a dataset, a runner, a scorer, and a threshold. Get these four pieces right and you have a working eval. Get any one wrong and your eval will either fail to catch regressions or drown you in false alarms.

The dataset

The dataset is a collection of input cases. Each case is an input to your system — a user message, a document, a query, a tool call — along with any reference data you need to score the output (the expected answer, a rubric, a set of required facts). The dataset should be representative of the distribution your system will encounter in production, weighted toward the cases that matter most.

Dataset quality is the primary determinant of eval quality. Fifty well-curated cases that represent real failure modes will catch more regressions than five hundred cases randomly sampled from happy-path logs. Invest in the dataset.

The runner

The runner is the code that feeds each input case to your system and captures the output. It should call your system the same way production calls it — same API client, same retry logic, same prompt construction code. Do not special-case the eval path. If your system has a bug that only manifests under real call conditions, the runner should hit it.

The scorer

The scorer takes a system output and a reference (if you have one) and produces a signal: pass/fail, a 0–1 score, or a structured rubric result. The scorer is the hardest part of the eval to get right. More on scoring strategies in the next section.

The threshold

The threshold is the aggregate score at which you consider the eval passing. 90% pass rate? 95%? Zero tolerance for certain failure categories? The right threshold depends on the stakes of the task. A customer-facing classification system tolerating 5% errors is a different risk profile than an internal document summarizer tolerating the same rate. Set the threshold explicitly. "It seems to be getting better" is not a threshold.

Scoring strategies

The right scoring strategy depends on what kind of output you are measuring. There is no universal scorer. Most production eval suites use multiple strategies for different parts of the same task.

Exact match

The simplest scorer. The output either matches the expected value or it does not. Use it for tasks where correct outputs are deterministic and enumerable: classification labels, boolean decisions, structured JSON fields, extracted values. Exact match is fast, cheap, and perfectly reproducible. The limitation is that it cannot handle cases where multiple outputs are equally correct, or where the output format can vary while the meaning is the same.

LLM-as-judge

You use a second model call to evaluate the output of the first. The judge model receives the input, the system output, optionally a reference answer, and a rubric, then returns a verdict. This is the most flexible scorer — it handles natural language quality, adherence to instructions, factual accuracy relative to a reference document, and tasks where the set of correct outputs is large or hard to enumerate.

LLM-as-judge requires careful prompt engineering for the judge itself. The judge's system prompt needs to specify exactly what "good" means for your task, include a concrete scoring rubric, and include worked examples of what each score level looks like. Without these, the judge will produce inconsistent scores that are hard to interpret. The judge model should be at least as capable as the model being judged — using a weaker model to evaluate a stronger one introduces systematic blind spots.

Human baseline

Human evaluation is expensive and slow but it is the only ground truth for tasks where quality is subjective. Use it to calibrate your automated scorers — run both human and LLM-as-judge on the same sample and measure how often they agree. If your LLM judge agrees with humans 85% of the time, you can use it at scale with known reliability. If it agrees 60% of the time, it is unreliable and you need to fix the judge before trusting its aggregate scores.

Embedding similarity

Compute the cosine similarity between the embeddings of the output and a reference answer. This captures semantic equivalence — two sentences that mean the same thing in different words will score high even with zero exact-match overlap. Useful as a complementary signal alongside exact match or LLM-as-judge, particularly for summarization and paraphrase tasks. The limitation is that embedding similarity is sensitive to topic overlap but insensitive to correctness — a plausible-sounding wrong answer can score high if it shares vocabulary with the reference.

Building a regression suite

A regression suite is an eval that you run on every change to catch regressions before they ship. Building one that stays useful over time requires discipline about what goes in and how you maintain it.

Start with failures

The best way to build a regression suite is to add a case every time you see a failure — in testing, in production logs, in user reports. Fix the bug, write the case that would have caught it, add it to the suite. Over time, the suite accumulates the failure modes that actually happen in your system rather than the failure modes you imagined during planning.

Version your datasets

Store your eval datasets in version control alongside your code. When you add, remove, or modify cases, the change is visible in the commit history. This lets you understand score changes over time — a drop in score might mean your system got worse, or it might mean you added harder cases. Without version control, you cannot tell which.

Run in CI

Add the eval to your CI pipeline so it runs on every pull request. This sounds expensive, but a well-designed eval suite for most production use cases runs in under five minutes against a few hundred cases. The cost of running the eval in CI is tiny compared to the cost of shipping a regression to production. Block merges on eval failures with the same priority you block on failing unit tests.

Segment your suite

Do not track only the top-line aggregate score. Segment your eval by case category — by difficulty tier, by input type, by the specific capability being tested. A system that improves overall but regresses on a specific edge case category is a system with a new bug that the aggregate score hides. Segmented results make the failure visible.

Eval anti-patterns

Evals fail in predictable ways. Most teams hit these eventually. Knowing them in advance saves the rework.

Golden datasets that drift

A golden dataset is a fixed set of input-output pairs that you treat as ground truth. The problem is that the correct output for a given input often changes over time — the product changes, the data changes, the definition of "correct" evolves. A golden dataset that is not updated drifts out of alignment with what you actually want the system to do. You end up with a high eval score that does not correspond to real-world quality.

The fix is to review your golden dataset periodically, especially after significant product changes. Audit a sample of cases, verify that the reference answers are still what you want, and update the ones that are not. Treat the dataset as a living artifact, not a fixed artifact.

Over-indexing on one dimension

When you optimize for a single eval metric, you often get a system that scores well on that metric and poorly on everything else. Optimize only for factual accuracy and you get responses that are accurate but unhelpful. Optimize only for fluency and you get responses that read well but say wrong things. Most useful outputs have several relevant dimensions. Measure several dimensions, set thresholds for each, and do not let improvement in one dimension mask regression in another.

Testing the wrong thing

The most insidious anti-pattern. Your eval suite passes and your production system still fails, because you are testing a different input distribution than production encounters. Common causes: the eval uses clean, well-formatted inputs while production inputs are messy; the eval covers the happy path but not the edge cases users actually hit; the eval tests the output format but not whether the content is correct.

The fix is to source eval inputs from production whenever possible. Sample real inputs from your logs, anonymize them, and add them to the dataset. This closes the distribution gap between what you test and what you deploy.

Eval gaming

Once you have an eval suite, there is a temptation to optimize the system specifically against the eval cases rather than against the underlying task. This is the machine learning equivalent of teaching to the test. The system learns to handle the specific inputs in your dataset and overfits to them, while the underlying problem remains unsolved.

The defense is a holdout set — a portion of your eval dataset that you do not run during development, only during final validation before a release. If the system scores well on the training eval but poorly on the holdout, you have overfit to the eval.