RAG vs fine-tuning: what businesses actually need

Almost every AI project we discuss with clients starts with the same question: "Should we fine-tune a model on our data?" Sometimes the answer is yes. More often, the real need is something else: an assistant that knows your current documents, cites its sources and can be updated on a Tuesday afternoon without a training run. This article explains the main options in plain language, when each one fits, and how to decide without burning months on the wrong approach.

The four tools in the box#

When people say "teach the model about our business", they usually mean one of four different techniques. They solve different problems, and they can be combined.

Retrieval-augmented generation (RAG)#

RAG leaves the model unchanged. Instead, when a question comes in, the system first searches your own content, such as manuals, contracts, tickets or a product catalogue, and passes the most relevant passages to the model together with the question. The model then answers based on those passages.

Three ideas matter here:

• Retrieval: finding the right passages. This is usually a mix of semantic search over embeddings and classic keyword search, often followed by a re-ranking step.
• Grounding: instructing the model to answer from the supplied material, and to say so when the material does not contain the answer.
• Citations: pointing to the exact document, page or passage an answer came from, so a human can check it.

RAG shines when knowledge changes often, when the corpus is large, and when people need to verify answers.

Fine-tuning#

Fine-tuning continues training a model on your own examples, so that its behaviour shifts. It is good at teaching how to respond: a consistent tone, a strict output format, a domain-specific classification scheme, or a narrow task performed thousands of times a day. It is a poor way to teach what is true right now. Facts learned through fine-tuning are hard to update, hard to trace back to a source, and can be blended or misremembered.

Fine-tuning also lets you move a narrow task to a smaller, cheaper and faster model, which can matter a lot at high volume.

Long-context prompting with prompt caching#

Modern models from Anthropic, OpenAI and several open-source families accept very long inputs. If your knowledge base is modest in size, for example a product handbook, a policy set or a set of FAQs, you can often put all of it directly into the prompt. No index, no retrieval pipeline, no chunking decisions.

The obvious objection is cost and latency: sending the same large document with every request is wasteful. Prompt caching addresses this. Providers can cache a stable prefix of the prompt, so repeated requests reuse the already processed content and are billed and served more efficiently. For a stable, bounded body of knowledge, long context with caching is frequently the simplest thing that works.

Structured outputs#

Many "AI" projects are really extraction projects: read an invoice, a CV, a contract or an email, and produce clean fields. Here the key capability is structured outputs, where the model is constrained to return data that matches a schema you define, for example JSON with specific fields and types. This is not about knowledge at all. It is about reliability of format, and it usually removes the need for fine-tuning in extraction tasks.

When each approach fits#

A useful way to think about it: separate knowledge from behaviour.

• Fresh or changing knowledge, and answers people must verify: use RAG, or long context if the material is small enough. Both let you update knowledge by updating documents, and both support citations.
• Stable, bounded knowledge that fits in the context window: start with long context and prompt caching. Move to RAG when the material outgrows it or when you need fine-grained access control per document.
• Consistent style, tone or format across many outputs: first try clear instructions and a few good examples in the prompt. If that is not enough at your volume, fine-tuning is a legitimate option.
• Narrow classification or routing at high scale: fine-tuning a smaller model, or simply using a smaller general model with a well-designed prompt, is often the most economical path.
• Extraction of fields from documents: structured outputs, optionally combined with document inputs and citations so each extracted value can be traced.

These are not exclusive. A mature system might use RAG for knowledge, structured outputs for the response format and a small fine-tuned model for routing incoming requests.

Side-by-side comparison#

A practical decision checklist#

Before choosing an architecture, answer these questions honestly:

1. What is the task, exactly? Answering questions, drafting text, classifying, or extracting? Write down five real examples of input and ideal output.
2. How often does the underlying knowledge change? Daily or weekly changes point strongly away from fine-tuning.
3. Do users need to verify answers? In legal, finance, compliance, support or healthcare contexts, citations are usually non-negotiable.
4. How large is the knowledge base? If it comfortably fits in a context window, try long context with caching first.
5. Who may see what? If different users have access to different documents, you need retrieval with permission filtering, not a single shared prompt or a model trained on everything.
6. What volume and latency do you expect? High volume on a narrow task is where smaller or fine-tuned models earn their keep.
7. Do you have labelled examples? Fine-tuning without a substantial set of good examples rarely beats a well-written prompt.
8. How will you measure success? If you cannot answer this, stop and build an evaluation set first.

If most answers point to "changing knowledge, needs citations, moderate volume", you want RAG or long context. If they point to "stable task, strict format, very high volume", consider fine-tuning or a smaller model.

Common pitfalls#

These are the problems we see most often when reviewing AI systems that "almost work".

• Bad chunking. Splitting documents into arbitrary fixed-size pieces cuts tables in half, separates headings from their content and loses context. Chunk along the document's structure and keep useful metadata such as title, section and date.
• No evaluations. Without a test set, every change is a guess. Teams tweak prompts, swap models and change chunk sizes without knowing whether things got better or worse.
• Stale indexes. The source documents were updated, the index was not. The assistant confidently quotes last year's policy. Re-indexing must be part of the content workflow, not a manual afterthought.
• Hallucination without citations. If the system does not show where an answer came from, users cannot distinguish a grounded answer from an invented one. Require citations and teach the model to say "I don't know" when the sources are silent.
• Privacy and GDPR. Personal data in documents, prompts and logs is still personal data. Know which provider processes it, in which region, under which data processing agreement, and how long logs are retained. Fine-tuning on personal data deserves extra caution, because removing it later is difficult.
• Prompt injection from documents. Retrieved content is untrusted input. A document can contain text that tries to instruct the model, for example to ignore its rules or to reveal other data. Treat retrieved text as data, limit what the model can do with tools, and never let document content grant permissions.

Evaluate before you optimise#

The single most valuable step in any AI project is also the least glamorous: build an evaluation set before you pick an architecture.

A good starting set is simply a few dozen to a few hundred real questions or inputs, each with the expected answer or the documents that should be cited. Then measure:

• Retrieval quality: did the system find the right passages?
• Answer quality: is the answer correct, complete and grounded in the sources?
• Citation accuracy: do the cited passages actually support the claim?
• Refusal behaviour: does it say "I don't know" when it should?
• Format compliance: for extraction, does every output match the schema?

With this in place, comparisons become factual. You can test long context against RAG, one model against another, or a fine-tuned model against a prompted one, on your own data rather than on generic benchmarks. You will often find that the cheapest fix is better retrieval or a clearer prompt, not a bigger model or a training run.

Automated grading with a model can speed this up, but spot-check it with human reviewers, especially early on.

How our own demos illustrate the patterns#

We try to practise what we recommend, and two demos on this site show the patterns in action.

The sigmacode Assistant answers questions about our services and approach. Its knowledge base is small and deliberately frozen, so instead of a retrieval pipeline it uses long-context prompting with prompt caching: the whole knowledge base sits in a cached prompt prefix, and the model is instructed to answer only from it. For a bounded body of knowledge, this is simpler to build and easier to keep correct than a vector index.

The Document Q&A demo shows the other side. You provide a document, ask questions, and each answer comes with citations pointing to the passages it relied on. That is the core promise of grounded AI: every claim can be checked against its source.

Neither demo required fine-tuning. That is typical. For most business knowledge problems, grounding and good evaluation matter more than custom training.

The short version#

• Use RAG when knowledge is large, changes often, needs access control, or must be cited.
• Use long context with prompt caching when the knowledge is stable and fits in the window.
• Use fine-tuning or smaller models for consistent behaviour or narrow tasks at high volume, not for facts.
• Use structured outputs for extraction and any output a system has to parse.
• Evaluate first, then optimise what the numbers tell you to.

If you are weighing these options for a real project, our AI & Automation team, led by a tech lead with 20+ years of experience, can help you scope it, build an evaluation set and ship a first version you can actually measure. Get in touch and tell us what you are trying to solve.