How to evaluate RAG: retrieval metrics, faithfulness and golden sets

A RAG system has two stages that fail in different ways. Retrieval can miss the passage that holds the answer. Generation can ignore, misread or embellish a passage it was given. If you only score the final answer, you learn that something went wrong without learning which half to fix.

This guide is for engineers and product owners running RAG chatbots, search assistants or document question answering who want measurements they can trust. It covers the two families of metrics, how to check faithfulness, how to build a golden set and where LLM judges mislead.

Read the metrics together. Low recall with high faithfulness means the model behaves well on poor context, so fix retrieval. High recall with low faithfulness means the right passages arrive and the model drifts from them, so look at the prompt, the model or how context is formatted. Correct answers with low faithfulness are a warning too: the model is answering from its training data, which breaks as soon as your documents say something different from general knowledge.

Retrieval metrics are cheap and deterministic once relevant passages are labelled, so run them on every change. Our embeddings and semantic search guide shows how to compute recall@k and nDCG.

Faithfulness asks whether the answer says only what the sources support. Scoring a whole answer at once is unreliable, because one unsupported sentence hides easily among correct ones. A more dependable method:

1. 01Split the answer into atomic claims, using a model, or sentence splitting for short answers.
2. 02For each claim, ask a judge model whether the retrieved passages support it, contradict it or say nothing about it.
3. 03Require the judge to quote its supporting evidence, and verify that the quote exists in the sources.
4. 04Report the share of supported claims per answer, and flag any contradicted claim.

import json
from anthropic import Anthropic

client = Anthropic()

JUDGE_PROMPT = """Decide whether the sources support the claim.
Reply with JSON only: {{"verdict": "supported" | "contradicted" | "not_found", "quote": "<exact text or empty>"}}

Sources:
{sources}

Claim:
{claim}"""

def judge_claim(claim: str, sources: list[str]) -> dict:
    context = "\n\n".join(sources)
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=300,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(sources=context, claim=claim)}],
    )
    result = json.loads(response.content[0].text)
    if result["verdict"] == "supported" and result["quote"] not in context:
        result["verdict"] = "unverified"  # the judge quoted text that is not in the sources
    return result

The quote check matters. Judges sometimes mark a claim as supported and invent the evidence, and a string match catches that cheaply. In production, normalise whitespace before matching and use structured outputs where available, so the JSON always parses.

A golden set is a curated collection of questions paired with what a correct outcome looks like. For RAG, each entry should hold:

• The question, written the way real users ask it.
• The identifiers of the passages that contain the answer.
• A reference answer, or the key facts a correct answer must include.
• Tags for intent, difficulty and risk, plus a flag for questions the documents cannot answer.

Source questions from search logs, support tickets and domain experts, and include unanswerable questions on purpose. Aim for a few hundred entries across your main document types. Label relevant passages at a level that survives re-chunking, such as document and section, so the set stays valid when you change your chunking strategy.

Version the golden set and review it when documents change. Otherwise a question whose answer moved to a new policy version will penalise the system for being right.

Model judges make answer evaluation practical at scale, and they have well-known failure modes:

• Length and style bias. Judges tend to favour longer, more confident answers. Grade against specific criteria and key facts, and avoid asking which answer is better overall.
• Self-preference. A judge may favour outputs from its own model family. Where you can, judge with a different model from the one that generates.
• Position bias. In pairwise comparisons, the first option can win more often. Run both orders.
• Vague rubrics. "Rate helpfulness from 1 to 10" produces noise. Use a few categories with written definitions.
• Unvalidated judges. Grade a sample by hand and measure agreement before trusting the judge, and check again after changing the judge model or prompt.
• Leakage. If the judge sees the reference answer while checking faithfulness, it may reward matching the reference instead of matching the sources.

1. 01Build a golden set of real questions with labelled passages, reference facts and unanswerable cases.
2. 02Run retrieval metrics on every change to chunking, embeddings, search or reranking.
3. 03Run faithfulness, correctness, citation and refusal checks on every change to prompts or models.
4. 04Validate the judge against human grades, and check again whenever the judge changes.
5. 05Gate releases on agreed thresholds, as described in LLM evals before every release.
6. 06Sample production conversations weekly and turn failures into new golden set entries.

We set up this kind of evaluation before tuning any RAG system, because it turns every later change into a measured decision. Our law firm RAG case is an example of a system built this way, the production RAG chatbot guide covers the build itself, and how we work explains where evals fit in our delivery.