LLM evals: how to test AI features before every release

An eval is an automated test for an AI feature. You run a fixed set of inputs through your system, score the outputs and compare the result with the previous release. This post is for product and engineering teams shipping LLM features who want to change prompts, models and retrieval without guessing what broke.

Traditional unit tests assume the same input gives the same output. LLM systems are probabilistic, and small changes travel far: a prompt tweak that fixes one complaint can quietly break twenty other cases, and a new model version can shift tone, format and refusal behaviour at once. Without evals, every release is a judgement call made on a few hand-picked examples.

With evals in place, you get:

• A quality score you can track over time.
• A release gate that blocks regressions before users see them.
• The confidence to swap models or cut cost, because you can measure the effect.
• A shared definition of "good" between product, engineering and domain experts.

The most useful eval sets come from real traffic and real work. Synthetic questions written by the team tend to be clean and predictable, while real users write short, ambiguous, misspelled questions and paste in half a document. Use synthetic cases to fill gaps, after the real ones.

Good sources for cases:

1. 01Production logs, sampled across user types and intents, with personal data removed.
2. 02Support tickets and escalations, especially the ones where the AI got it wrong.
3. 03Domain expert examples for rare cases that matter a lot, such as a claim that must be rejected.
4. 04Adversarial cases: prompt injection attempts, out-of-scope questions and inputs in other languages.

Start with 50 to 100 cases and grow to a few hundred. Tag each case by intent, difficulty and risk, so you can see when a release improves simple questions while regressing on high-risk ones. Every bug found in production should become a new case before it is fixed.

Use the cheapest check that reliably catches the failure. Deterministic checks are fast, free and repeatable. Model-graded checks handle qualities that code cannot judge, but they are slower, cost money and need validating themselves.

In practice we combine them. Code checks run first and fail fast. A judge model then scores what remains against a short rubric, with a reference answer where one exists. Before trusting a judge, grade 50 or so cases by hand and compare. If the judge often disagrees with your experts, fix the rubric before you gate releases on it. Our post on evaluating RAG covers judge pitfalls in more depth.

Evals earn their keep when they run automatically. We run a fast suite on every pull request that touches prompts, tools, retrieval or model configuration, and the full suite before each release. A minimal runner looks like this:

import json, statistics, time
from dataclasses import dataclass

@dataclass
class Case:
    id: str
    input: str
    must_include: list[str]
    must_not_include: list[str]

def run_suite(cases: list[Case], answer_fn, threshold: float = 0.92) -> None:
    passed, latencies = 0, []
    for case in cases:
        start = time.perf_counter()
        output = answer_fn(case.input).lower()
        latencies.append(time.perf_counter() - start)
        ok = all(s.lower() in output for s in case.must_include) and not any(
            s.lower() in output for s in case.must_not_include
        )
        passed += ok
        if not ok:
            print(json.dumps({"case": case.id, "output": output[:200]}))
    score = passed / len(cases)
    p95 = statistics.quantiles(latencies, n=20)[18]
    print(f"pass rate {score:.1%}, p95 latency {p95:.2f}s")
    if score < threshold:
        raise SystemExit(1)

A few rules make the gate trustworthy:

• Compare with the baseline as well as an absolute threshold, so a drop from 96% to 93% is visible even when 93% passes.
• Mark critical cases that must always pass, whatever the overall score.
• Run non-deterministic cases more than once and take the majority result, so one noisy output does not block a release.
• Store every run with the prompt version, model id and commit hash, so you can trace any change in score.

A release that adds two points of accuracy and doubles the cost per request may not be a good release. Record input and output tokens, cost per case and latency percentiles on every run, next to the quality scores. A model change then becomes a trade-off you can discuss with numbers: for example, a newer model might score higher on high-risk cases while being slower and more expensive, and the eval report shows by how much.

Set budgets as part of the gate, such as a p95 latency limit and a maximum cost per 1,000 requests. Once you can prove quality holds, simple intents can often move to a smaller, cheaper model.

Evals cover what you anticipated. Production shows you everything else. Run a small, regular human review loop next to the automated suite:

• Sample a fixed number of production conversations each week, weighted towards low-confidence, negative-feedback and high-risk cases.
• Have domain experts grade them with the same rubric the judge uses, which also tells you how closely the judge agrees with people.
• Turn every confirmed failure into a new eval case.

Review time stays modest, and the eval set keeps pace with how people actually use the product.

1. 01Pull 50 real cases from logs or tickets and write down the expected outcome for each.
2. 02Add deterministic checks for format, required fields and forbidden content.
3. 03Add one model-graded check with a short rubric, and validate it against human grades.
4. 04Run the suite in CI on prompt, model and retrieval changes, with a threshold and critical cases.
5. 05Log cost and latency on every run.
6. 06Start a weekly review sample and feed failures back into the set.

We set up evals at the start of every AI project, before the first prompt is tuned, because they make every later decision faster. Our claims assistant case shows what weekly releases on an eval gate look like, and how we work explains where evals sit in our delivery process.