Written by the people who caused the bugs
These are notes from running a shopping assistant on real stores: the bugs we caused, how we found them, what we measured and what we changed. They are short, specific and written by the person who fixed the problem. If a note gives a number, the number came from a measurement we can reproduce, and if it did not hold on a second store, the note says so.
The notes, newest first
-
A bug that only appeared on colour products: attribute terms are display labels, variations record slugs, and comparing the two directly fails silently.
-
A bug where product cards contradicted the text above them, and why choosing cards from retrieval rather than from the answer was the wrong instinct.
-
Most reported hallucinations are one diagnosable state (retrieval returning zero candidates) and the threshold behind it is a number you can measure.
About these notes
Why we publish our bugs
Every AI chat vendor promises accurate answers. Almost none will tell you how their answers go wrong. We think the second is more useful to you than the first. A note about a product card that showed a pickle when the answer was about powder tells you more about how carefully a system is built than any claim on a home page. It also tells you what to watch for on your own store, whichever vendor you choose.
What you will find here
- Retrieval failures that look like hallucinations and are not: the right passage was never retrieved, or was retrieved and scored below the threshold.
- Store API quirks that silently break a shopping assistant, such as WooCommerce reporting the same option in two alphabets.
- Measurements of thresholds, chunk sizes and costs, with the method, so you can repeat them.
- Design decisions behind the widget, such as why it loads a few kilobytes before anyone opens it.
A few things we have learnt
A wrong answer is far more often a missing or badly written page than a model inventing something. The same threshold behaves differently on two stores with different writing styles. Store data is messier than it looks: options arrive in two spellings, sizes are sold as separate products, and prices hide in page titles. Each of those became a note here, and each changed the product.
Who writes them
The engineer who found and fixed the problem writes the note, usually within a day or two of the fix, while the details are still fresh. Notes are read through by a second person before they go up, to check that every figure is reproducible and that nothing in the note overstates what the fix does. When a later change makes an older note out of date, the note says so at the top.
How to follow along
Subscribe by RSS: no account, no email address and nothing that tells us you are reading. New notes are also listed in the changelog when they describe a change to the product.
The product these notes are about
Every measurement here came off the trace panel, which is the part of the product worth looking at first.