Grounding Language Models on Proprietary Data: Retrieval That Doesn't Drift
Language models hallucinate on internal data because they were never trained on it. Grounding them reliably is a retrieval problem, not a model problem. Here's how we approach it.
Ask a general-purpose language model about your business and it will answer confidently and often wrongly. Not because the model is weak, but because it has never seen your data. It was trained on the public internet; your contracts, your client records, your internal processes were not in that corpus. When you ask it a question that requires them, it does the only thing it can — it produces the most plausible-sounding answer, which is not the same as the correct one.
Closing that gap is the single most common problem in applied AI, and the instinct most people reach for — a bigger, smarter model — is the wrong tool. Grounding is a retrieval problem before it is a model problem.
The model is only as good as what you put in front of it
The pattern that works is retrieval-augmented generation: before the model answers, you fetch the relevant pieces of your own data and place them in the model's context, then instruct it to answer from those pieces and not from memory. Done well, the model stops guessing and starts summarising real source material. Done badly, it is a confident liar with citations.
The quality of the whole system is set at the retrieval step, not the generation step. If you fetch the wrong three documents, the best model in the world will give you a fluent, well-written wrong answer. If you fetch the right three, a modest model will give you a correct one. This is why we spend most of our engineering effort on retrieval quality and comparatively little on model selection — the leverage is lopsided, and it points at the part most teams under-invest in.
Chunking is where quality is won or lost
Retrieval works by breaking your documents into pieces, indexing them, and pulling back the pieces most relevant to a query. How you break them up — chunking — quietly determines everything downstream.
Chunk too large and each piece contains several ideas, so a query matches on one sentence and drags four irrelevant paragraphs into the context, diluting the signal. Chunk too small and you sever the context a sentence needs to make sense, so the model receives a fragment it can't interpret. There is no universal right answer; the correct chunking depends on the structure of your specific documents, and finding it is empirical work — you test, you measure retrieval quality, you adjust. It is unglamorous, and it is the difference between a system that works and one that merely demos.
Hallucination is a failure of instruction as much as retrieval
Even with perfect retrieval, a model will sometimes answer from its training rather than from the documents you gave it, because that is its default behaviour. Suppressing this is partly a matter of explicit instruction — tell the model to answer only from the provided sources, and to say plainly when the sources don't contain the answer — and partly a matter of making refusal an acceptable outcome.
That last point matters more than it sounds. A system that is allowed to say "I don't have that information" is far more trustworthy than one engineered to always produce something. In regulated and high-stakes settings, a confident wrong answer is worse than no answer, because someone acts on it. Building a model that knows the edge of its own knowledge — and stops there — is a design choice you have to make deliberately, because the technology's instinct runs the other way.
Every answer should be traceable to a source
The final discipline is citation. When the system answers, it should be able to point at the specific documents the answer came from, so a human can verify it. This is not a nicety. It is what turns a black box into an auditable tool, it is what lets you debug a wrong answer by seeing which source misled it, and it is increasingly what separates AI you can deploy in a serious business from AI you can only demo.
Grounding, in the end, is a matter of engineering humility. The model is a powerful reasoning and summarising engine, but it does not know your business, and pretending otherwise is where the failures come from. Feed it the right facts, instruct it to stay within them, let it admit when it can't, and make every claim traceable. Do that and you get something rare: an AI system that is not just fluent, but reliable — reliable enough to build on.