Chunking

Chunking is the step in a retrieval-augmented system where documents are split into passages, or chunks, small enough to be retrieved individually. It happens once, when content is added, and it quietly sets the ceiling on how good every later answer can be.

Why documents are split at all

A search that returns a whole thirty-page policy document has not really found anything: the model still has to find the one sentence that answers the question, and the document may not even fit. Splitting it into passages lets the search return the paragraph about international delivery rather than the entire terms of sale.

The trade-off

  • Chunks that are too large mix several topics. A passage about both delivery times and return windows matches questions about either, and scores only moderately for both, so the right answer can lose to a worse but more focused passage.
  • Chunks that are too small lose their context. "Within 7 days" means nothing without the heading that says it is about returns, and a passage that has lost its heading can be retrieved for the wrong question.

Splitting on structure

The better approach is to split where the document itself changes subject: at headings, list items and table rows, rather than every so many characters. A heading stays attached to the paragraph it introduces, and a table row about one product does not bleed into the next.

Products are their own chunks

In a shopping assistant, each product should be exactly one chunk, carrying its name, price, stock, options, image and link together. A product split across two chunks can be recommended with the wrong price, and two products merged into one chunk can be confused with each other.

How to tell chunking went wrong

In a trace, a chunking problem looks like the right information retrieved under the wrong context: a passage that contains the answer but also contains three other things, or one that contains the answer without the heading that makes it make sense. The fix is to restructure the page, not to change the model.

A quick test

Pick a page and read one passage from it out of context. If you cannot tell what it is about, the assistant cannot either.

How ChatWidget does it: documents are split on their structure, each product is one passage, and you can preview exactly what every passage contains before the assistant uses it.

Related terms

  • Retrieval-augmented generation (RAG)

    Retrieval-augmented generation, usually shortened to RAG, is a way of making a language model answer from specific documents instead of from whatever it absorbed during training. When a question arrives, the system first retrieves the passages most likely to contain the answer, then asks the model to generate a reply using only those passages.

  • Embedding

    An embedding is a list of numbers that represents the meaning of a piece of text. Passages that mean similar things get similar lists of numbers, even when they share no words at all. That is what lets an AI assistant search by meaning instead of by matching words.

  • Similarity threshold

    The similarity threshold is the minimum score a retrieved passage must reach before an AI assistant is allowed to use it in an answer. Everything above the line can be used; everything below it is dropped. It is the single setting that decides whether an assistant answers a question or declines it.

  • Deflection rate

    Deflection rate is the share of conversations that an automated assistant handled without passing them to a person. If 1,000 people start a chat and 700 never reach a human, the deflection rate is 70 per cent. It is one of the most quoted numbers in customer service software, and one of the easiest to flatter.

  • Resolution rate

    Resolution rate is meant to be the share of conversations in which the customer's problem was actually solved. It sounds like the right thing to measure, and it is, which is why it matters so much how each vendor defines "solved". Two vendors quoting the same percentage can be measuring completely different things.

See it on a real answer

The trace is where the vocabulary becomes a line between what was used and what was dropped.

See the trace panel All terms