RAG vs fine-tuning vs prompt engineering: pick the layer, not the religion
RAG, fine-tuning and prompt engineering solve different problems and combine in practice. Prompt engineering changes only the input, RAG adds current company data at runtime, fine-tuning permanently retrains the model weights. The decision matrix scores them on freshness, effort, cost, hallucination risk and maintenance. Pexon builds RAG platforms and treats prompt engineering as the default starting point.
The decision matrix
Prompt engineering, RAG and fine-tuning, scored on the questions that decide
| Criterion | Prompt engineering | RAG | Fine-tuning |
|---|---|---|---|
| What changes | Only the input (the prompt) | Context added at runtime | Model weights, permanently |
| Data freshness | Limited to model knowledge | Current data from the knowledge base | Depends on training data |
| Effort | Low | Medium: pipeline and indexing | High: training and evaluation |
| Cost | Token costs only | Retrieval plus token costs | Training plus hosting |
| Hallucination risk | Medium | Lower: answers carry source context | Medium |
| Maintenance | Low | Ongoing indexing required | Regular retraining required |
| Typical use | Simple tasks, style, instructions | Company knowledge, support, document search | Domain vocabulary, stable special cases |
Read the table as a stack, not a contest: each technique covers a gap the others leave open, which is why production systems combine them. The rows are the decision criteria; the columns are the levers.
The rule the matrix defends
Prompt engineering is the default starting point: cheapest, fastest, reversible. Add RAG when the answer depends on current or company-internal data. Fine-tune only when the task needs a stable style or vocabulary that prompts and retrieval cannot reach — and only after measuring the gap.
Prompt engineering: the layer you already have
Prompt engineering changes only the input. The model stays untouched, the data stays untouched, and the improvement comes from writing better instructions, examples and context into the prompt. The effort is low, the effect on well-defined tasks is often large, and the change is reversible in seconds: if the new prompt is worse, the old one comes back.
Its limit is the model's own knowledge. A prompt cannot add a document that was written after the training cutoff, cannot know the contents of your contracts, and cannot cite a source it has never seen. When the question is about company-internal or current data, prompt engineering alone is not enough — which is exactly where the next layer starts.
The practical rule: measure the prompt baseline before you buy anything. A good prompt engineer on a mid-size model often closes most of the gap that a fine-tuning project would be bought to close, at a fraction of the cost, and the measurement makes that visible instead of arguable.
RAG: the layer for knowledge that changes
RAG stands for retrieval-augmented generation. Before the model answers, a retrieval system searches your documents and database for the relevant passages, and those passages are handed to the model as context alongside the question. The model itself does not change; the knowledge base does, and that is the point.
The advantages follow from that architecture. Data freshness: updating the index updates every future answer, no retraining needed. Citations: the answer can point at the record it came from, which is the difference between a helpful assistant and one an auditor can verify. Control: the access rules of your organisation can be mapped into the retrieval layer, so the model sees only what the user may read.
The cost is the pipeline: chunking, embeddings, the vector store, retrieval tuning and the ongoing indexing of new documents. That is why RAG projects live or die on data preparation and evaluation rather than on model choice — the retrieval quality decides the answer quality, and it is the part nobody can buy as a subscription.
Fine-tuning: the layer for stable specialisation
Fine-tuning retrains a base model on your data, permanently changing its weights. The result internalises a style, a domain vocabulary or a task shape, so the model needs shorter prompts and produces more consistent output for that specific job. It is the right tool when the pattern is stable and the data is well-defined.
The costs are the reason the threshold is high. Training runs cost compute and engineering time; evaluation costs a test set and a process; hosting a specialised model costs infrastructure, especially if you run it yourself; and when the data changes, the model needs retraining. The maintenance line is what most estimates forget.
The decision rule that keeps fine-tuning honest: if the task changes less than the data, fine-tune; if the data changes less than the task, retrieve. RAG wins on moving knowledge, fine-tuning wins on a fixed style — and the gap between them is where most real deployments sit, which is why the combination is the production norm rather than an edge case.
Our position: the question is not 'RAG or fine-tuning'. It is which layer carries which part of the answer, and the answer is usually all three — with prompt engineering as the base, RAG as the knowledge layer, and fine-tuning reserved for the specialised last mile.
Why the combination is the norm in production
A production assistant for an industrial company shows the stack working together. Prompt engineering shapes every request: the instruction set, the output format, the guardrails. RAG supplies the knowledge: the contracts, the engineering documents, the quality records, retrieved per query and cited in the answer. Fine-tuning, where it exists, adds the vocabulary and tone the other two cannot reach — the way this company names its products and describes its processes.
The engineering consequence is that the three layers have different failure modes and different owners. Prompt drift breaks the format; retrieval gaps break the facts; a stale fine-tune breaks the style. Each needs its own evaluation, and the evaluation is what the decision matrix is really for: it tells you which layer to tune when the answer quality moves, instead of re-arguing the architecture.
The cost logic follows the same stack. Prompt engineering is token-cost; RAG adds the retrieval pipeline and its maintenance; fine-tuning adds training and hosting on top. The honest budget order is the same as the build order — prompts first, RAG second, fine-tuning only when the numbers say so — and the measurement that supports the decision is a test set with a recorded baseline, which is the discipline we build into every engagement.
Questions we get asked when the prototype works and the production choice is due
What is the difference between RAG and fine-tuning?
RAG adds relevant documents as context at runtime without changing the model, so answers can cite current company data. Fine-tuning permanently retrains the model weights on your data, which suits stable, recurring patterns rather than changing information. RAG is the better default for company knowledge; fine-tuning earns its cost only for stable specialisation.
When should I use RAG instead of fine-tuning?
Whenever the data changes, the volume is large, or citations are required. RAG retrieves the relevant passages at query time, so the knowledge base can be updated without retraining. Fine-tuning is the better fit when the task is a stable style or vocabulary and the training data is well-defined and durable.
Can RAG, fine-tuning and prompt engineering be combined?
Yes, and in practice the combination is the norm. Prompt engineering is the base layer on every request, RAG supplies current and company-internal context, and fine-tuning adds a special style or domain vocabulary where the other two cannot reach. Each technique covers a different gap, so they stack rather than compete.
What does fine-tuning cost compared to RAG?
Prompt engineering costs only tokens. RAG adds the retrieval pipeline, the index and its maintenance on top of token costs. Fine-tuning is the most expensive: training runs, evaluation and ongoing hosting of a specialised model, plus regular retraining when the data changes. The honest rule is to start with prompts, add RAG for knowledge, and fine-tune only when the cost is justified.
Next step
Start with prompts, add retrieval, fine-tune only when the numbers say so
Fixed-price entry sprint: we build the prompt baseline and the retrieval layer for one use case, measure answer quality against your own test set, and hand you the decision with the evidence attached. The sprint is yours to keep whether or not we build the platform.