What is RAG?
RAG, short for retrieval-augmented generation, is a way to make AI answer from your own documents. When someone asks a question, the system first searches your documents for the passages that answer it, then gives those passages to a language model, which writes the answer and cites where each part came from.
A language model is the kind of AI behind chat assistants: software trained on large amounts of text that reads what you give it and writes a reply. It knows nothing about your internal procedures, contracts or manuals. RAG hands it the relevant pages when a question comes in.
How RAG works
RAG works in two stages. Once, and again whenever documents change, the system prepares your documents: it reads them, splits them into passages and stores each passage in a searchable form. Then, for every question, it finds the most relevant passages, gives them to the language model with instructions and returns an answer with citations.
Preparing the documents
- Ingest. Connectors collect documents from drives, intranets or a document management system and extract the text. Scanned pages need OCR (text recognition) first.
- Split. The system cuts each document into passages of a few paragraphs, ideally along its headings, so each passage covers one topic.
- Embed. Each passage is turned into an embedding and stored in a search index with its source, date and who may read it.
An embedding is a list of numbers that represents what a passage means, so passages about the same subject sit close together even when they use different words. Most systems also keep a keyword index, because search by meaning can miss exact terms such as a clause number.
One question, from start to answer
- 01An employee asks: "Can I carry unused annual leave over to next year?"
- 02The system checks which documents that employee may read.
- 03It searches the index by meaning and by keyword, among those documents only, and keeps the best few passages, such as the leave section of the HR policy.
- 04It sends the model the question, those passages and instructions: answer only from these passages, cite each one you use, and say so if they do not contain the answer.
- 05The model writes the answer. The system checks that every citation points to a retrieved passage, then shows the answer with links to the sources.
Permissions are applied in the search itself: each passage carries the access rights of its source document. Once a restricted passage reaches the model, the model can repeat it, whatever its instructions say.
RAG examples from daily operations
RAG fits wherever people spend time looking things up in documents that change, and need an answer they can check against the source. The three examples below are illustrative, typical cases, each with the documents involved and what a person still checks.
Staff questions about policies and procedures
For example, a care organisation keeps HR policies and work instructions on an intranet, some in several versions. A RAG assistant answers questions such as "Which form do I use to report an incident?" from the current procedures only, with a link to the page. Procedures for managers are retrievable only by managers. Questions the documents do not answer get a clear "not found", and HR can see which ones keep coming up.
Support answers from product documentation
A typical case: manuals, release notes and known-issue articles are spread over a help centre and an internal wiki. Inside the helpdesk, RAG retrieves the passages for the customer's product and version and drafts a reply with links. The support employee checks, edits and sends it. The version filter matters: an older release's manual gives answers the customer knows are wrong.
Contracts and case files, with citations
For example, a procurement team asks "What did we agree with this supplier about late delivery?" The answer sits in a contract, perhaps amended by an addendum. RAG returns the answer with the exact clause and page, so the buyer reads the source before relying on it. The same works for case files at an insurer or a municipality.
Questions about every document, such as "list all contracts with a price indexation clause", are a poor fit, because RAG only reads the few passages it retrieves. For those, a better approach is to extract the clauses into a table first and query the table.
Retrieval inside a process
Docket, our reference build, checks supplier invoices against a written procurement policy. It retrieves the policy clauses that apply to each invoice, decides, and cites the clause behind every automatic decision. Invoices that don't pass go to a person, with the reason already written out.
When to use RAG, fine-tuning or long context
Use RAG when answers must come from many or changing documents, users need to check the source, or different people may see different documents. Put documents straight into the prompt (long context) when the set is small and stable. Fine-tuning trains a model further on examples, which shapes how it writes more than what it knows.
| Aspect | RAG | Long context | Fine-tuning |
|---|---|---|---|
| What it does | Sends the model only the passages relevant to each question | Sends whole documents with every question | Changes how the model responds, through extra training |
| Document set | More than a model can read at once | Small enough to fit in one prompt | None; facts it picks up can't be traced |
| When documents change | Re-index the changed documents | Send the new version | Retrain the model |
| Citations | Built in | Possible, if you ask the model to quote | Not available |
| Per-user access | Filtered in the search | You pick the documents per user | Not available |
| Running cost grows with | The passages sent per question | The full length of the documents, every time | Hosting the model and retraining it |
| Good fit | Policies, contracts, knowledge bases, product documentation | One handbook or one contract | A fixed output format, a house style or a narrow task at high volume |
The approaches combine, and our article on RAG vs fine-tuning covers that choice in detail. If the questions are about data in a system, such as order status or stock levels, a query against that system beats all three.
What goes wrong with RAG, and how to catch it
Most RAG failures start before the model writes anything. Retrieval misses the passage that holds the answer, or returns an outdated version, a badly parsed table or a passage the user should not see. The model then answers from what it was given, or invents an answer when nothing useful came back.
Retrieval misses the right passage
Search by meaning returns passages on the right topic that never mention the product code asked about. Or a rule and its exception were split into two passages and only the rule came back. Splitting along headings, adding keyword search and retrieving neighbouring passages help.
Stale and duplicate documents
The same procedure sits in three folders: last year's version, the current one and a draft. Retrieval finds all three, and the model quotes the old one or blends them. Index only approved, current versions, and remove passages when a document is replaced.
Tables and scans parsed badly
A basic text extractor turns a price table into loose numbers, losing which price belongs to which product. A scanned PDF without text recognition produces no text at all, and the document silently drops out of every answer. Check a sample of each document type after extraction.
Permissions that leak
Leaks happen when access rights are copied into the index once and never synced, so someone removed from a folder keeps getting answers from it, or when the prompt is trusted to hide content. Filter in the search, sync rights from the source systems, and test with an account that should see nothing.
The model still invents an answer
When retrieval finds nothing relevant, the model may answer from its general training anyway, citing nothing or citing a passage that says something else. Instruct it to say when the passages do not contain the answer, check citations in code, and show "not found" when an answer has no valid citation.
Measure it with a test set
Collect real questions from future users, each with the passage that answers it, plus questions the documents cannot answer. Score retrieval and answers separately: is the right passage in the top results, and is the answer correct and supported by what it cites? Rerun it after every change. Our guide to evaluating RAG retrieval quality shows how.
Getting started: a ready-made tool or a custom build
Start with a ready-made "chat with your documents" tool when one department asks questions about one source that everyone may read, and people check the answers. Have RAG built when answers draw on several systems, permissions differ per user, citations must hold up in an audit, data must stay in the EU, or answers belong inside your own software.
When a ready-made tool is enough
Many office suites and AI assistants let you upload files or connect a drive and ask questions. That is enough when:
- The documents sit in one place and rarely change.
- Everyone who uses it may read all of them.
- People check the source before acting on an answer.
- Where the provider stores and processes the files fits your data rules.
When to have it built
- Answers draw on several sources, each needing its own connector and sync.
- Access differs per person or team, and the search has to follow the rights in each source system.
- Citations must be auditable: months later you can show which passage, from which version, supported an answer.
- Documents and questions have to stay in the EU, through an EU-hosted model provider or your own cloud account.
- Answers belong inside the systems people already use, such as a ticket or a case file.
Our guide to building a RAG system people can rely on covers the build layer by layer. Our RAG starter, open source on GitHub, is the structure we begin document Q&A builds from. To have it built and run for you, see how we build RAG systems that answer from your documents with citations and permissions.
Questions about RAG
Is RAG the same as a chatbot?
No. A chatbot is an interface where people type questions and get replies. RAG is a method for grounding answers in your documents, and it can sit behind a chat window, a search box, a draft reply in a helpdesk or a step in an automated process. A chatbot without RAG answers from the model's general training.
Does RAG stop AI from making things up?
RAG makes made-up answers less likely, and they still happen. The model answers from passages you supply and cites them, so a reader can check the source. The model can still misread a passage, combine two sources wrongly or fill a gap when retrieval finds nothing relevant.
So add checks around the model: a citation check in code, a "not found" response and a test set. For decisions that matter, a person reads the cited source.
Can our documents stay in the EU?
Yes. In the RAG systems we build, the documents, search index and logs can run in EU regions, and the model runs through an EU-hosted provider or inside your own cloud account. With a ready-made tool, check this before you upload anything.
What is the difference between RAG and search?
Search returns documents or passages, and the reader finds the answer in them. RAG searches first, then has a language model write an answer from the top passages, with citations. When people need the whole document, such as a template or a form, good search is enough.