How to build a production RAG chatbot: a practical guide

This guide is for teams that have a RAG chatbot working on a laptop and need it to work for a whole organisation. Retrieval-augmented generation (RAG) answers questions from your own documents: it finds the relevant passages, passes them to a language model and asks for an answer grounded in them. A first version that answers a handful of questions takes an afternoon. A version that answers thousands of questions a week, from documents that change, for users with different permissions, takes engineering.

In production, a RAG chatbot has to:

• Find the right passage for the question, including exact terms such as product codes, clause numbers and names.
• Cite its sources so a user can check the answer in seconds.
• Respect permissions, so nobody sees an answer built from a document they cannot open.
• Say when it does not know, instead of filling gaps with plausible text.
• Stay measurably good as documents, prompts and models change.

The rest of this guide walks through each layer in the order we build them.

Retrieval can only return what ingestion extracted. PDFs with two-column layouts, scanned pages, tables split across pages and headers repeated on every page all turn into noisy text when you run them through a basic text extractor. The model then answers from fragments, and no amount of prompt work fixes that.

A solid ingestion pipeline does four things well:

1. 01Parse by document type. Use layout-aware parsing for PDFs, keep table structure as Markdown or HTML, and run OCR or a vision model only on pages that need it.
2. 02Strip boilerplate. Remove repeated headers, footers, page numbers and disclaimers that would otherwise match every query.
3. 03Keep metadata. Store the source URI, title, section path, version, date, language and the access groups that can read the document.
4. 04Sync incrementally. Hash each document, re-index only what changed and delete chunks when the source is deleted. Stale chunks are a common cause of confidently wrong answers.

Treat ingestion as a pipeline with logs and retries, running on a schedule or on change events from the source system. For messy inputs such as invoices and forms, see our guide to document parsing with LLMs.

Chunking decides what a single retrievable unit looks like. Too small and a chunk loses the context that makes it meaningful. Too large and it dilutes the embedding and fills the context window with irrelevant text.

We usually start with structure-aware chunks of roughly 300 to 800 tokens, prefix each with its title and section path, and keep a pointer to the parent section. Then we let the evaluation set decide. Chunk size is a parameter to measure, and the best setting for a contract archive is rarely the best setting for a support knowledge base.

Vector search is good at meaning: a question about "ending a contract early" finds a clause titled "termination for convenience". It is weaker on exact tokens such as ISO 27001, an article number or a SKU. Keyword search with BM25 has the opposite profile. Hybrid search runs both and merges the results, which is why it is our default for business documents.

If your data already lives in Postgres, you can do this with pgvector and built-in full-text search, merged with reciprocal rank fusion (RRF). Postgres full-text ranking is similar to BM25 in spirit; where exact BM25 scoring matters, add a BM25 extension or a search engine next to the database.

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks
  WHERE tenant_id = $2 AND access_groups && $4
  ORDER BY embedding <=> $1
  LIMIT 50
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rank
  FROM chunks, plainto_tsquery('english', $3) AS q
  WHERE tenant_id = $2 AND access_groups && $4 AND tsv @@ q
  ORDER BY ts_rank_cd(tsv, q) DESC
  LIMIT 50
)
SELECT c.id, c.content, c.source_uri,
       COALESCE(1.0 / (60 + s.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS score
FROM semantic s
FULL OUTER JOIN keyword k ON s.id = k.id
JOIN chunks c ON c.id = COALESCE(s.id, k.id)
ORDER BY score DESC
LIMIT 20;

Note that the tenant and permission filters sit inside both searches. Filtering after retrieval can leave you with too few results, and it means restricted text passes through code that should never see it.

A reranker then scores the top 20 to 50 candidates against the question, using a cross-encoder or an LLM, and keeps the best five to ten. Reranking is often the cheapest large improvement in answer quality, at the cost of some added latency.

Ask the model to cite chunk IDs for every claim, then check the citations in code before the answer reaches the user:

• Every cited ID must be one of the chunks you actually passed in.
• Quoted text must appear in the cited chunk, after normalising whitespace.
• An answer without citations is replaced by a clear "I could not find this in the documents" response, and the case is logged for review.

These checks are deterministic, cheap and catch a large share of fabricated references. For deeper checks on whether an answer is actually supported by its sources, see how to evaluate RAG.

Access control belongs in retrieval. Store access groups on every chunk at ingestion time, keep them in sync with the source system, and pass the user's groups into every search. Do not rely on the prompt to hide restricted content.

A RAG chatbot fails in two places: retrieval misses the right passage, or generation misuses a passage it was given. Measure both. Build a set of 100 to 300 real questions with the documents that answer them, and track:

• Retrieval: recall@k and nDCG, which tell you whether the right chunks are in the top results.
• Answers: correctness against a reference answer, faithfulness to the retrieved sources and citation accuracy.
• Refusals: whether the chatbot declines questions the documents cannot answer.
• Operations: p95 latency and cost per conversation.

Run the set on every change to chunking, prompts, models or rerankers, and block releases that regress. Our post on LLM evals before every release covers the release gate in more detail.

1. 01Collect 100 real questions from the people who will use the chatbot, with the documents that answer them.
2. 02Build ingestion for your two or three most important sources, with metadata and permissions.
3. 03Start with structure-aware chunks and hybrid search, and measure recall@10.
4. 04Add a reranker and citation checks, then measure again.
5. 05Put the eval set in CI before you add more sources.
6. 06Launch to one team, review a sample of conversations each week and feed failures back into the set.

This is the order we follow when we build RAG systems for clients, from ingestion through to running them in production. Our law firm RAG case shows the result, and how we work explains how we run projects like this.