Eval Pipelines

3 min · workflow

An eval pipeline is an automated system for measuring model output quality. It runs on a dataset of inputs with known-good outputs (or grading criteria), scores the model's responses, and produces a metric. The pipeline belongs in CI — just like unit tests.

Running Evals in CI

Evals run at three cadences depending on cost and signal value:

On every PR — fast evals on a small golden set (50–200 examples). Block merge if score drops below threshold. Catches regressions introduced by prompt edits.
Nightly — full eval suite on the complete dataset. More expensive; run when traffic volume makes per-PR full runs cost-prohibitive.
Before production deploy — final gate. Same full suite, but with a human sign-off step if scores are within a margin of the threshold.

# GitHub Actions example
- name: Run evals
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    python evals/run.py \
      --dataset evals/data/golden_set_v3.jsonl \
      --model claude-opus-4-5 \
      --threshold 0.88

Exit non-zero on threshold failure to block the CI step.

Dataset Management

The dataset is a first-class artifact. Treat it like code.

Version it — store eval datasets in the repo or a content-addressable store. Every eval run records which dataset version was used.
Golden sets — a curated subset of high-signal examples that covers critical behaviors. Keep the golden set small enough to run cheaply on every PR.
Drift detection — production inputs shift over time. Sample live traffic periodically and compare its distribution against the eval dataset. High divergence means your evals are testing yesterday's distribution.
Failure cases — when the model fails on a production example, add it to the eval dataset before fixing the prompt. This creates a regression test.

LLM-as-Judge Setup

For tasks without a deterministic correct answer (summarization, tone evaluation, reasoning quality), use a separate model as the judge.

JUDGE_RUBRIC = """
Score the response on these criteria (1-5 each):
- Accuracy: Does it correctly address the question?
- Conciseness: Is it appropriately brief without omitting key details?
- Format: Does it follow the required output format?

Return JSON: {"accuracy": int, "conciseness": int, "format": int, "overall": float}
"""

def judge(question: str, reference: str, candidate: str) -> dict:
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=256,
        system="You are an objective evaluator. Return only JSON.",
        messages=[{
            "role": "user",
            "content": f"Question: {question}\nReference: {reference}\nCandidate: {candidate}\n\n{JUDGE_RUBRIC}"
        }]
    )
    return json.loads(response.content[0].text)

Use a different model (or smaller variant) for the judge than the model being evaluated. Same-model judging introduces systematic bias. Calibrate the judge against human scores before relying on it as a gate.

When to Block Deploys

Define thresholds before you have results, not after. Post-hoc threshold setting is rationalization.

Typical gating strategy:

Hard block: score drops more than 5% relative from the previous baseline on any critical category
Soft block (human review required): score drops 1–5% relative
Pass: score within 1% of baseline or improved

Track score history in a simple store (SQLite, a JSON file in the repo, a dashboard). Score trends over time are as important as point-in-time values.

See fine-tuning for why evals are required before any fine-tuning run.

Sources

linked from

AI in CI/CD