AI in CI/CD

3 min · workflow

Integrating model API calls into CI/CD pipelines adds automated quality checks that catch regressions prompt changes can introduce. The key discipline: put deterministic, automatable checks in CI, and keep human-judgment calls out of it.

What Belongs in CI

Evals — run a scored test suite against a golden dataset on every PR. If the score drops below threshold, block the merge. This is the most important AI step in CI. See eval-pipelines for setup.

Regression tests — specific named failure cases from production. Confirmed bugs get a test case before the fix lands, so they can't silently re-appear.

Prompt lint — static checks on system prompt files: token count within budget, no placeholder text left in, required sections present. Cheap to run, catches obvious errors.

Output schema validation — for apps with structured outputs, generate model responses on a sample input set and validate them against the defined schema. Catches prompt changes that break output format.

What Doesn't Belong in CI

Anything requiring human judgment on output quality — CI can check whether a score is above a threshold; it can't evaluate whether a response is actually good. A failing eval should require human review before it becomes a blocking gate.

Long-running creative evaluations — evaluating tone, creativity, or nuanced accuracy on open-ended tasks doesn't fit in a PR check. Run these on a nightly cadence with human sampling.

Real user data — CI pipelines should run on synthetic or anonymized datasets. Don't build a pipeline that calls the model API with production user inputs for every PR.

GitHub Actions Setup

name: Eval CI

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/**'

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python evals/run_ci.py --threshold 0.88 --dataset evals/golden_v4.jsonl

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: evals/results/

Rate Limits and Costs

The Anthropic API has per-minute and per-day token limits. CI pipelines can hit these on busy repositories.

Mitigations:

Use smaller models for CI evals when possible (Haiku for format checks, Sonnet for semantic evals)
Apply prompt-caching to system prompts that don't change between eval runs — the cache key persists across requests within the same day
Cap eval dataset size for PR checks; run full suites nightly
Add exponential backoff and retry logic to handle rate limit errors without failing the build

Costs: budget $0.10–$2.00 per CI run depending on dataset size and model tier. Track via the Anthropic usage dashboard and set billing alerts.

Practical Pipeline Structure

PR opened →
  ├── Prompt lint (< 5s, no API calls)
  ├── Schema validation on sample inputs (< 30s, 10-20 API calls)
  └── Golden set eval (2–10 min, 50–200 API calls)
        ├── Pass → merge allowed
        └── Fail → PR blocked, report posted as comment

Keep CI eval latency under 10 minutes to avoid developer friction. If evals run longer, cut the golden set or move to nightly.

eval-pipelines