RAG without illusions: why your company chatbot lies and how to fix it

Almost every company that approaches us about a knowledge base has the same attempt behind them: they uploaded a few PDFs to a chatbot, asked two questions, were delighted — and on the third question the system invented, with absolute confidence, a number that never appeared in the documents.

That is not a model failure. That is missing architecture. Let us look at what exactly is missing.

How RAG works (and where it breaks)

RAG stands for retrieval-augmented generation. The principle is simple:

1. Documents are split into chunks and converted into vectors.
2. The question is converted into a vector the same way.
3. The most similar chunks are retrieved.
4. The model receives the question together with the retrieved chunks and writes an answer.

Step 4 is where it breaks. The model is instructed to “answer based on the context” — but if the answer is not in the context, most models will write one anyway. Not out of malice: generating text is their job and silence is not the default behaviour.

The first thing that separates a demo from a system: the ability to say “I do not have this in the sources.”

This is not a prompt setting you can toggle. It is a combination of thresholding on retrieval scores, an explicit instruction and — above all — a test set that also contains questions with no answer. If your test set has no questions where the system should say “I do not know”, you will never find out that it cannot.

The second thing: citations are not cosmetic

An answer without a source is unusable in a company. Not because it is necessarily wrong, but because it cannot be verified. When someone asks “based on what?”, the answer “the AI said so” is not an answer.

The practical impact is bigger than it sounds. When a system cites the document and the page:

• a user can verify the answer in ten seconds,
• incorrect answers get reported instead of quietly spreading,
• and you get feedback you can actually act on.

Citations also change the model's behaviour. When the system is obliged to name a concrete source, its willingness to invent drops sharply — simply because there is nothing to cite.

The third thing: permissions, or compliance stops the project

This is where an otherwise well-run project most often stalls. It looks innocent: let us index everything and let people search.

The problem: that “everything” includes payroll data, disciplinary records and contracts with commercial terms not everyone should see. Without permission-aware retrieval you build a system that will happily disclose anything to anyone.

The fix is not complicated but must be planned upfront: every chunk carries information about who may see it, and filtering happens during retrieval, not at display time. Filtering at the end is a security hole — the model has already seen the content and can paraphrase it.

An audit log belongs with it: who asked what and what they received. Not because you want to monitor employees, but because without it you cannot prove anything.

Where quality is actually won

Surprisingly, not on the model. The order of impact as we see it in practice:

1. The state of the documents. If you have three versions of the same policy and two contradict each other, no model will resolve that. The first phase of any project is an inventory and it tends to be uncomfortable — commonly a third of the content turns out to be out of date.

2. Chunking. A chunk must be big enough to make sense on its own and small enough that several fit into context. A table split in half is worse than no table.

3. Hybrid retrieval. Vector search alone fails on exact terms — contract numbers, product codes, names. Combining it with classic full-text search is one of the cheapest improvements available.

4. Reranking. Retrieving twenty candidates and letting a smaller model pick the five most relevant is usually markedly better than taking the first five.

5. And only then the model. Yes, a bigger model phrases things slightly better. But if retrieval returned poor material, not even the best model will extract truth from it.

How you find out whether it works

Take 50 to 150 real questions from your people. For each one, write the correct answer and the document containing it. Add at least ten questions for which the answer does not exist.

Then measure three things:

• Factual accuracy — how many answers are true.
• Correct “I do not know” — how often the system admitted it had no answer.
• Citation accuracy — how often it pointed at a genuinely relevant source.

Without these numbers, any claim about quality is just an impression. With them you can say whether a change helped — and that is the whole difference between engineering and guessing.

Summary

The difference between a demo and a usable knowledge base is not the model. It is that the system can say “I do not know”, can show its source, and respects who may see what. The first two require a test set. The third requires that someone thought about it before the first line of code.

If this is what lies ahead of you, enterprise AI knowledge base is exactly the service where we tackle these three things first.