A version of this conversation happens every few weeks. A business has thousands of documents — contracts, policies, product specs, past tickets — and wants an assistant that knows them. The instinct is to fine-tune a model on the documents. It sounds right. It is almost always wrong, and the wrongness is expensive because you only discover it after the training run.
Why fine-tuning on documents disappoints
Training absorbs patterns, not facts. Feed a model ten thousand contracts and it becomes fluent in the register and shape of your contracts. It does not reliably recall clause 7.2 of the agreement you signed in March. Worse, it will produce something clause-shaped and confident, because that is exactly the pattern it learned.
Then the documents change. A price list updates, a policy is revised, a contract is amended — and the knowledge is baked into weights that now need retraining. Retrieval handles the same update by reindexing one file in seconds.
What each one is genuinely for
In practice, sixty to eighty percent of private knowledge problems are solved with retrieval alone, and a good share of the rest are solved by better prompting before any training is justified.
The ladder we actually climb
We work through these in order and stop at the first rung that hits the accuracy target. Each step up multiplies cost and reduces how quickly you can change your mind.
- Prompting — a properly specified system prompt, structured outputs, sensible temperature. Costs nothing. Skipped far too often.
- Retrieval — parse, chunk, embed, index, retrieve. Where most of the value is, and where most of the engineering craft lives.
- Adapter fine-tuning — small trained matrices layered onto a frozen base model. Cheap enough to run on modest hardware, and each client keeps a separate adapter over one shared base.
- Preference tuning — training on pairs your experts ranked, once you have real judgements about what good looks like.
- Continued pretraining — domain-adaptive training on a large corpus. Genuinely justified sometimes. Rarely.
Retrieval is where the actual difficulty is
“Just add RAG” hides a pile of decisions that determine whether the system is useful or merely demonstrable:
- Parsing — scanned PDFs, merged table cells, multi-column layouts and headers that repeat on every page
- Chunking — splitting on structure rather than character count, so a clause is not severed mid-sentence
- Hybrid search — keyword retrieval alongside vectors, because exact identifiers and part numbers do not embed well
- Reranking — the first twenty candidates are rarely in the right order
- Permissions — retrieval must respect who is asking, or you have built a very efficient data leak
- Evaluation — a held-out question set, scored, so “it feels better” becomes a number
That last one is the difference between a prototype and a system. Without an evaluation set, every prompt change is a guess, and nobody can tell whether last week's fix broke something else.
The shape that usually wins
Retrieval for knowledge. Light tuning only where behaviour genuinely drifts. Tools for anything that touches a real system — the CRM, the database, the scheduler. Human review on anything consequential, routed by a confidence threshold rather than by hope.
That architecture is unglamorous and it holds up. Most systems that fail in production failed because someone skipped straight to training, or because nobody built the evaluation harness that would have caught the regression.