Every platform in this category demos well. They are all built on similar models and they all answer your straightforward questions correctly, which means a demo tells you almost nothing about month three.
These are the questions that do separate them. Several of them are awkward for us too, and those are marked.
Can you show me a wrong answer, and which stage produced it?
Not a good one. Bring a question you know is hard, something with a number, an exception, or a condition, and ask the vendor to show you what the system did when it got it wrong.
Then ask: can you tell me whether the right document was retrieved?
If the answer is no, understand what you are accepting. Every future quality problem will be diagnosed by guesswork: rewrite a source, add an instruction, try again. Three months in, teams typically have a long list of instructions, some contradicting others, and no way to reason about behaviour.
In a review of ten platforms in July 2026, nine gave no visibility into chunking, embedding or the vector store. It is the single biggest differentiator in the category and almost nobody asks about it during a trial.
What exactly does the billing unit count?
Get this in writing, with edge cases:
- Does opening the widget count?
- Does the greeting?
- Does a turn where the assistant declined to answer?
- Does a turn that errored?
- Do messages you send yourself while testing?
- Does re-indexing your content?
Two of the ten platforms we assessed publish no pricing at all, and one returns an error page where pricing should be, whose body has caused automated tools to invent rate cards from it. If a vendor will not put the unit definition in writing, assume the definition favours them.
Then ask what happens when you exceed it: does the assistant stop answering, or does it keep going and bill you? Both are defensible. Discovering which one you bought from an invoice is not.
What are the limits nobody publishes?
These decide whether a platform works for you, and they are almost never on the pricing page:
| Limit | Why it matters |
|---|---|
| Tool timeout | If your store takes six seconds and the timeout is five, live stock never works. |
| API rate limit | Decides whether you can drive it from your own systems. |
| Version history depth | Whether you can roll back a bad configuration next week or only today. |
| Environments | Whether you can test a change before customers see it. |
| Trace or log retention | How far back you can investigate a complaint. |
What do your deflection and resolution rates actually measure?
Two numbers dominate these conversations and both can be made to look better than reality.
Deflection rate counts conversations that did not reach a human. It improves when the assistant gets worse: a visitor who believes a wrong answer does not ask for a person, they leave, and sometimes they buy the wrong thing. An assistant with no handover button deflects everything.
Ask specifically: does an abandoned conversation count as deflected? Very often it does.
Resolution rate is the number you actually want, and it cannot be derived from transcripts alone. It needs an outcome signal, a feedback press, a completed purchase, an absence of a follow-up. Every proxy is imperfect. Ask which signal underlies the figure, and if the answer is a feedback button, ask what share of conversations produce one. It is usually low single digits.
What is not certified, and when was the last penetration test?
Request the current status of SOC 2, ISO 27001, and the date of the last external penetration test.
The useful part is the shape of the answer. A vendor who says “we are aligned with SOC 2 principles” does not hold a report. One who describes their encryption when asked about certification is changing the subject. A vendor who tells you plainly what they do not have is easier to trust on what they do.
This one is awkward for us. We hold no SOC 2 report, no ISO 27001 certificate, and have not yet commissioned an external penetration test. If any of those is a hard requirement this quarter, we are not your answer, and we would rather you knew now, the security page says so above the table rather than below it.
Where does our data go, at index time and at query time?
Storage region is the question everyone asks. The one that gets missed:
What is transmitted at index time, as distinct from query time? If a vendor computes embeddings using a third-party service, your entire corpus is leaving their infrastructure, not only the passages relevant to one question. Those are very different disclosures and only one of them usually appears on a trust page.
Also ask whether your content or your visitors’ conversations are used to train models. Get it in the contract rather than the FAQ.
Also awkward for us. We store data in India and compute embeddings ourselves, but generating an answer sends the question and the retrieved excerpts to a model provider that may process them in the United States. That hop is inherent to a hosted model, and a vendor claiming single-region while using one is describing storage and calling it processing.
How does the assistant know who a visitor is?
If you plan to let the assistant look up an order, ask how it knows who it is talking to. A signed token from your own backend is authentication. “The widget says they are logged in” is not, a claim that is not cryptographically signed can be forged by anyone editing a page.
Awkward for us again: we have no signed visitor identity yet, which is why our order-lookup guidance asks for an order number and an email the way a support agent would.
What does the widget cost our page, measured?
For a website widget, ask for the loader size gzipped, what loads before a visitor opens the panel, and the contribution to Cumulative Layout Shift. Then ask whether those figures are enforced anywhere or merely stated.
Watch for a budget quoted as a measurement. “8 KB” might be a ceiling rather than a reading, we made exactly that mistake on our own landing page and corrected it. Ask for the measured number and the limit separately.
What does the roadmap say you cannot do today?
The tenth question is the one that separates a vendor from a brochure. A roadmap that names what is missing, in the order it is coming, is a vendor telling you where you would be disappointed before you are. A page with no gaps on it has not been read by a customer yet.
The ten, on one page
- Show me a wrong answer, and tell me which stage produced it.
- Define the billing unit, in writing, with edge cases.
- Tool timeout, rate limit, history depth, environments, trace retention.
- Does an abandoned conversation count as deflected?
- What signal underlies your resolution rate?
- What is not certified, and when was the last penetration test?
- What leaves your infrastructure at index time, not only at query time?
- How does the assistant know who a visitor is?
- Loader size measured, and is the budget enforced?
- What does the roadmap say you cannot do today?
If a vendor answers all ten plainly, that is informative regardless of the answers. Our own attempt is on the roadmap and the comparison pages, and where a competitor is genuinely better we say so first.
Found something here that is out of date, or wrong? Tell us, a guide with a review date on it is a promise to keep it current.