How LLM evals let an insurance claims assistant ship weekly at 94% accuracy

94%answer accuracy, tested on every release

Four hundred graded cases run on every change, so a better prompt never quietly breaks an answer that used to be right.

The problem

Every change to the claims assistant risked breaking answers that used to be right, so releases slowed to once a quarter.

An evaluation suite of four hundred graded cases that runs on every change, with tracing to show exactly which answers moved and why.

The assistant ships weekly at 94% answer accuracy, tested on every release.