The short answer
If you want a language model to answer from your company's documents, data or policies, you almost always need retrieval-augmented generation (RAG). If you want a model to behave differently, such as following a strict output format, writing in a house style or doing a narrow classification task cheaply, fine-tuning can help. Many teams reach for fine-tuning first because it sounds like teaching the model about the business, then find it handles facts poorly.
This post is for business and product leaders deciding how to customise an LLM, and for engineers who need to explain the trade-off. We cover what each approach does, a decision table, when to combine them and what each costs to run.
What RAG and fine-tuning each do
RAG leaves the model unchanged. At question time, your system searches your documents, puts the most relevant passages into the prompt and asks the model to answer from them. Update a document and the next answer reflects it. Every answer can cite its source, and access control can be applied per user at search time.
Fine-tuning changes the model's weights by training it further on examples of inputs and desired outputs. The model picks up patterns: format, tone, terminology and how to handle a type of task. It does not reliably learn facts in a way you can update, cite or restrict, and whatever it does absorb is frozen at training time.
A useful way to think about it:
- RAG changes what the model knows at the moment it answers.
- Fine-tuning changes how the model tends to respond.
- Prompting, with clear instructions and examples, shifts both a little and costs the least. Try it first.
A decision table
| Your requirement | RAG | Fine-tuning |
|---|---|---|
| Answers from internal documents, policies or product data | Strong fit | Poor fit |
| Content changes weekly or daily | Strong fit: re-index what changed | Poor fit: needs retraining |
| Answers must cite sources | Strong fit | No built-in support |
| Different users may see different content | Strong fit: filter at retrieval | No built-in support |
| Strict output format or house style | Usually achievable with prompting | Good fit when prompting is not enough |
| High-volume narrow task, such as classification | Adds little | Good fit: a smaller tuned model can be cheaper and faster |
| Specialist vocabulary the base model handles badly | Helps by putting definitions in context | Can help, given enough high-quality examples |
For most business use cases, such as internal knowledge assistants, customer support, policy checks and document question answering, the rows at the top of the table dominate. That is why RAG is our default starting point, with fine-tuning considered later for specific, measurable reasons.
When to combine RAG and fine-tuning
The two approaches work together. Combining them makes sense when retrieval is already working and a measurable behavioural problem remains. For example:
- A support assistant retrieves the right articles, but answers must follow a strict structure that prompting does not hold consistently.
- A high-volume pipeline classifies documents before retrieval, and a small fine-tuned model handles that step at a fraction of the cost of a large general model.
- An embedding model or reranker is fine-tuned on your own query and document pairs, so retrieval understands your terminology better.
The last option is often overlooked. Fine-tuning the retrieval components can improve a RAG system more than fine-tuning the generating model, because retrieval quality sets the ceiling on answer quality. Our guide to embeddings and semantic search covers that side.
Cost and maintenance
The build cost is only part of the picture. Over a few years, what matters is how much effort each approach takes to keep correct.
| Cost area | RAG | Fine-tuning |
|---|---|---|
| Upfront work | Ingestion, chunking, search, citations and permissions | Collecting and cleaning hundreds to thousands of labelled examples, plus training runs |
| Keeping content current | Incremental re-indexing, largely automated | Rebuild the dataset and retrain |
| Model upgrades | Swap the model and rerun the evals | Retrain on the new base model, or stay on the old one |
| Running cost | Search infrastructure and longer prompts | Hosting or per-token pricing for the tuned model |
| Debugging | Inspect the retrieved passages to see why an answer happened | Harder, because behaviour is spread across the weights |
The model upgrade row deserves attention. General models improve quickly, and a RAG system can usually adopt a better model by changing configuration and rerunning the eval set. A fine-tuned model ties you to its base model until you invest in retraining.
Both approaches need an owner after launch. For RAG, that means watching ingestion jobs, reviewing failed questions and keeping the eval set current. For fine-tuning, it means curating new training examples as the task drifts and deciding when a retrain is worth the effort.
A practical decision guide
- 01Write the problem down as outcomes: which questions or tasks, for whom, and how you will judge a good answer.
- 02Try strong prompting with a capable general model and a small eval set, and measure.
- 03If answers need your knowledge, build RAG and measure retrieval and answers separately, as described in how to evaluate RAG.
- 04If a specific behaviour still fails after that, and you can collect good examples, fine-tune for that behaviour and compare the results on the same eval set.
- 05Revisit the decision when base models change. Something that needed fine-tuning last year may not need it now.
When we scope AI work with clients we follow this sequence, and most projects end up with RAG and good evals. Our production RAG chatbot guide goes deeper into the build, and you can book a call to talk through your own case.