Your chatbot did not hallucinate. It retrieved nothing.

Every wrong answer gets reported the same way: “the AI made it up”. Almost none of them are that.

A model asked a question with no supporting material in its context will produce something plausible, because that is what it is for. The failure happened one stage earlier, when retrieval returned nothing worth passing on. From the outside those two look identical. From inside the trace they look nothing alike, and only one of them is fixable by editing a prompt.

The number in the middle

Retrieval scores every candidate chunk by similarity and cuts at a threshold. Set it too high and real content is discarded; too low and a blog post about a spring sale gets treated as a pricing source. The threshold is therefore the most consequential number in a RAG system, and it is usually chosen by feel.

We measured ours on a grocery catalogue of 88 products:

product queries  ("do you have pickles", "dosa mix", "masala powders")   0.36 – 0.55
support queries  ("return policy", "track my order", "talk to a human")  0.12 – 0.28

There is a gap between those bands, and the threshold belongs in it. We set 0.32. Before that measurement it sat at 0.45, inside the product band. So “what pickles do you have” scored 0.36, landed under the cut, and returned nothing at all. The assistant then answered from general knowledge and invented a product line.

Nobody would have found that by rewriting instructions. The instruction was fine.

What our numbers do not tell you

A caution about those bands: they are one catalogue, in one language, in one product category. A store selling industrial fasteners by part number will produce a different distribution, and a threshold tuned on groceries may sit in the wrong place entirely. The method transfers. The constant does not.

Measure it on your own content, and measure again when the content changes shape.

The state worth naming

When retrieval clears nothing, the trace says so:

0 chunks above threshold

That line is the most diagnostic thing this product prints. It means the model had no material, not that it reasoned poorly. The fix is a content fix or a threshold fix, and both take minutes once you know which one you are looking at.