An assistant can only answer from what it can find. Most disappointing deployments are not model failures or prompt failures: they are content failures, decided before anyone typed a question.
This is what to do first, in rough order of how much difference it makes.
Write down the answers that only exist in someone’s head
Start here, because it is the highest-value work and the least obvious.
Every business has a set of questions that get answered dozens of times a week and are written down nowhere: what the actual lead time is in December, whether you ship to a particular region, what happens if a customer wants to change an order after dispatch. Support staff know them. The website does not say them.
An assistant indexing only your published pages will confidently guess at every one. Half a day writing those into a plain document and uploading it will improve answer quality more than any other single thing you can do.
Understand what a chunk is, because it changes how you write
Documents are split into passages and retrieved one at a time. That has a practical consequence: a passage has to make sense on its own.
Consider a returns policy written like this:
Returns are accepted within 30 days.
This does not apply to sale items.
Those may land in different chunks. Retrieve the first and the assistant tells a customer they can return a sale item. The exception was two paragraphs away and might as well have been in another document.
Written as one passage, the problem disappears:
Returns are accepted within 30 days, except on sale items, which are final.
The general rule: keep a condition in the same paragraph as the claim it qualifies. This is also only better writing for humans, which is why it is a comfortable change to make.
Give each fact one home
If your delivery charge appears on four pages, and three are out of date, retrieval will sometimes find the stale one. It is not doing anything wrong: it has no way to know which page you maintain.
Pick one page per fact and make the others link to it. Duplication is the commonest cause of an assistant that is right on Monday and wrong on Thursday.
Check what the crawl could actually reach
Content behind any of these is invisible to a crawl:
- A login or paywall.
- A
robots.txtrule, or anoindextag. - JavaScript that fetches the text after the page loads.
- An image of text, a PDF price list that is a scan rather than text is a picture as far as indexing is concerned.
- A PDF nobody has linked from anywhere.
A good crawl reports each page as fetched, skipped or failed with a reason. Read that list once. It is usually where you discover that your most useful page was never included.
Structure a product catalogue so each product answers about itself
If you sell things, a synced catalogue beats a crawl of your category pages, because each product becomes a passage carrying its own name, price, stock, image and link. Retrieve it and you have everything needed to answer.
Two things worth checking in your own store data:
- Prices for products sold in several sizes. If the store reports a single price for a variable product, that figure is usually the cheapest variation. An assistant quoting it as “the” price will understate the large size, and a customer will notice at checkout.
- Titles containing markup. Store platforms return titles with HTML entities in them more often
than you would expect. If
Mahani Pickle – 300greaches a customer verbatim, that came from your catalogue, not from the model.
Delete more than you add
The instinct is to index everything. Resist it. Old campaign pages, superseded policies, a blog post announcing a sale that ended in April: each is a candidate that can outscore the page you meant to be authoritative.
If a page would embarrass you as an answer, it should not be a source.
Then test with the questions that have numbers in them
Prices, dates, dimensions, lead times, eligibility. Vague questions are simple to answer plausibly; specific ones are where content gaps show. Ask the ten your customers actually ask, and read the trace on each rather than only the answer.
If a trace shows no passages retrieved, that is a missing document, not a bad model. It is also the cheapest bug you will ever fix.
A rough order of work
- Write down the unwritten answers. Upload as a plain document.
- Read the crawl report; fix what could not be reached.
- Remove sources you would not want quoted.
- Merge duplicated facts into one page each.
- Rewrite any policy where the exception is separated from the rule.
- Test with ten specific questions and read the traces.
Steps one and two are most of the value. Step six is how you find out whether you were right.
Found something here that is out of date, or wrong? Tell us, a guide with a review date on it is a promise to keep it current.