RAG or fine-tuning: which one, and when

RAG and fine-tuning don’t solve the same problem. RAG gives a model information; fine-tuning teaches it behaviour. For most enterprise projects RAG is the right starting point: when a document changes you re-index instead of retraining, and every answer can cite its source. Fine-tuning earns its place when output format, tone or domain-specific reasoning has to be locked in. In one agriculture case study the gains stacked: over 6 points of accuracy from fine-tuning, roughly 5 more from RAG.

They answer different questions

The debate is usually framed as “which is better”. Wrong question. The right one: is the thing your model lacks a fact, or a skill?

A language model has two kinds of memory: the parametric memory baked into its weights during training, and the non-parametric memory — an external store reached at runtime, whose contents are carried into the context window. Lewis and colleagues built RAG on exactly this distinction in 2020 — attaching an external memory, reached by a retriever at runtime, to a pre-trained generation model.

What RAG does

A question arrives, your own document store is searched, the most relevant passages are retrieved, they are placed into the prompt, and the model is asked to answer from those passages alone. The model learns nothing; it is simply handed the right pages every time. The chain typically runs: document → chunking → embeddings → vector/hybrid search → reranking → prompt → answer with citation.

What fine-tuning does

Fine-tuning updates a pre-trained model’s weights using your examples. You show it thousands of “given this input, respond like this” pairs, and it internalises the pattern.

In practice today this rarely means full training. Parameter-efficient methods dominate. LoRA (Hu et al., 2021) freezes the original weights and injects small, trainable low-rank matrices into each layer. The figures reported in the paper: up to 10,000× fewer trainable parameters and 3× lower GPU memory requirements compared to full fine-tuning of GPT-3 175B with Adam. That moved fine-tuning out of “only giants can do this” territory — but cheap is not the same as correct.

Comparison table

Criterion	RAG	Fine-tuning
Cost	Moderate setup (vector database + indexing). In operation every query carries a longer prompt → ongoing token cost	Training cost looks one-off but recurs with every data refresh. In operation the prompt is short → cheaper per query
Freshness	High. Index the document; it reaches answers the same minute	Low. New knowledge means a new training cycle. In between, the model states stale facts with full confidence
Data privacy	Data stays in the index, not in the model. Per-user permissions can be enforced at query time	Data is absorbed into the weights. Honouring a deletion request realistically means retraining. Authorisation cannot live inside the model
Setup time	Days. A rough prototype comes up fast; the real work is data hygiene and retrieval quality	Weeks. The bottleneck isn’t GPUs — it’s producing labelled examples and an honest evaluation set
Hallucination control	Strong. Answers are grounded in retrieved passages, sources can be cited, “I don’t know” is enforceable. But if retrieval is poor, the model will confidently summarise the wrong passage	Weak. The model sounds more fluent and more like you — including when it’s wrong. No verifiable source
Choose it when	Knowledge changes, citations are required, data is large and permission-scoped, domain expertise already exists in documents	Format/schema is rigid (e.g. every answer a specific JSON), tone matters, domain jargon or reasoning is needed, latency and token budgets are tight

The row most often overlooked is data privacy. Once a fact is inside the weights, taking it back out is not straightforward. With RAG you delete a record from an index; with a fine-tuned model you are left holding a model shaped by that data. Where the model runs is a separate question again: if data leaves for an overseas provider, Turkey’s KVKK cross-border transfer regime — in force since 1 June 2024, with standard contracts and binding corporate rules — applies. Running models on the organisation’s own infrastructure often closes that discussion before it starts.

The hybrid approach is the actual default

Treating these as alternatives is what destroys the most value. In Balaguer and colleagues’ agriculture study the two contributions added up: fine-tuning alone improved accuracy by more than 6 percentage points, and RAG added roughly 5 points on top. In the same work, once the fine-tuned model drew on information from across geographies, answer similarity on location-specific questions rose from 47% to 72%.

The logic is clean: fine-tuning teaches the model how to do the job; RAG tells it which data to do it with.

The right order for a hybrid build:

Start with RAG. You only learn where the value is — and where the model stumbles — from a system that’s actually running.
Keep an error log. Sort wrong answers into two piles: (a) the right document wasn’t retrieved → a retrieval problem; (b) the right document was retrieved and the answer was still bad → a behaviour problem.
If (a) dominates, don’t fine-tune. Fix chunking, hybrid search and reranking. That’s where the gain is.
If (b) dominates, you have already accumulated exactly what fine-tuning needs: real questions and the answers they should have had.
Train a small model with LoRA and place it inside the RAG chain. Freeze the evaluation set before training.

The sequence isn’t arbitrary. Without a running RAG system you cannot produce good fine-tuning data in the first place.

Common misconceptions

“Fine-tuning is how you teach a model new facts.” The most expensive misconception. Fine-tuning does transfer knowledge, but inefficiently and without an audit trail. Soudani and colleagues found RAG surpasses fine-tuning by a large margin precisely on factual knowledge about less popular entities. “Let’s teach the model our staff directory” is almost always a RAG job.

“RAG eliminates hallucination.” It doesn’t; it makes it controllable. If retrieval surfaces the wrong passage, the model will summarise it fluently. RAG quality is mostly retrieval quality; model choice is the remainder.

“Context windows got huge, so RAG is obsolete.” Pushing an entire corporate archive into every prompt is unsustainable on cost, latency and access control alike. The point was never “give it everything” — it’s give it the right thing.

“Fine-tuning makes the model smarter.” Fine-tuning aligns behaviour; it doesn’t add capability. Training on narrow data can erode a model’s general abilities.

“We’ll fine-tune first and add RAG later.” Backwards. The training data fine-tuning needs is born from the error logs of a working RAG system.

How we help at Albatros

An example from production, for a sense of scale: LEGAPALAS, which we build end to end, is a legal AI tied to more than 11 million court decisions. RAG carries the factual layer — every answer is grounded in a verifiable source.

We don’t make this call for you on a whiteboard: we stand up a small RAG system on your own documents first, measure where it breaks, and put fine-tuning on the table only when the measurements call for it. Where data cannot leave the organisation, we run the models on your own infrastructure. If you’d like to talk through where you stand, get in touch.