What actually breaks in RAG

Everyone has the same demo. You take a PDF, split it into chunks, stuff the chunks into a vector store, and ask a question. The answer comes back with a citation. The room nods. Then you load the real corpus and the whole thing quietly falls apart.

I have built and debugged a few of these systems now, and I can tell you where the time actually goes. It is never the model. The model is fine. The model will happily answer from whatever garbage you hand it. The garbage is the project.

Retrieval is the whole product

Think about what happens between the question and the answer. A user types something ambiguous, your system embeds it, finds a handful of chunks that look similar, and pastes those chunks into a prompt. The generation step is almost trivial. The hard part is that the right chunk exists, is findable, and is actually the right chunk when you find it.

Most teams spend a week wiring up the pipeline and then discover that the retrieval quality is the thing they have to tune forever. Chunk size, chunk overlap, whether to split on section boundaries, whether the title of the source document should be prepended to every chunk, whether metadata filters are applied before or after the similarity search. Each one moves the answer quality more than swapping to a better model does.

The uncomfortable truth is that retrieval quality is mostly an information architecture problem. The model can only answer from what it is given. If your source documents are a pile of PDFs with no structure, no consistent naming, no metadata, no sense of which document is authoritative when two of them disagree, no retrieval system will save you. You will ship a system that is confidently wrong about documents nobody has read.

Chunking is where your recall lives or dies

The default chunking strategy is to split on a character count with a little overlap, which is a bit like cutting a book into pieces every five hundred characters and hoping each piece still makes sense. It works well enough for paragraphs, badly for tables, and terribly for anything where context spans pages.

A few things I have learned the hard way:

Split on structure when you can. Headings, sections, and lists are natural boundaries. A chunk that starts and ends mid-sentence is a chunk that produces mid-thought answers.

Keep the provenance with the chunk. The chunk should know its document, its section, its page, its position. You cannot cite a source you did not store. Most grounding failures I have seen are not model failures. They are chunks that forgot where they came from.

Think about what the user will actually search for. If your documents are legal contracts, the query will be "termination notice period", not a sentence that happens to sit near the termination clause. The chunk that contains the answer needs to be retrievable from the language of the question, and that often means writing metadata, synonyms, and titles that the pure text does not contain.

Tables and numbers need special handling. Embeddings are terrible at structured data. A table of prices embedded as free text is nearly useless when someone asks "what is the price for 500 units of X". Sometimes you need to keep structured data in structured form and let the model query it, instead of flattening it into prose and hoping similarity search finds it.

Grounding is a promise you have to keep

The whole point of RAG is that the model should not answer from memory. It should answer from your documents. But "should" is doing a lot of work in that sentence. Unless you enforce it, the model will happily mix its training knowledge with your sources, and the user will not be able to tell which is which.

The fix is to make the answer provably traceable. Every claim in the response should be attached to a source that actually contains the claim. I am not talking about a pleasant-looking citation. I mean a system that checks, after generation, whether the cited chunk really supports the sentence it is attached to. This is the difference between a demo and a product. When a hospital system tells a clinician something, or a payment system tells a merchant something, the citation is not decoration. It is the entire trust model.

The cheap version of this is a claim-level check: split the answer into sentences, and for each one, verify that the supporting chunk actually contains the substance. It is not a perfect guard, but it catches the worst failure mode, which is a fabricated answer wearing a citation as a costume.

Evaluation is not optional, it is the project

Here is the pattern I keep seeing. A team gets a RAG pipeline working on a handful of curated examples. Everyone is excited. Then a stakeholder asks the obvious question: how well does this actually work? And nobody can answer, because nobody built a test set.

You need a test set. Not one. You need a hundred or so realistic questions with known answers and known source documents, and you need to measure two separate things. First, did retrieval find the right chunk? That is a retrieval metric, and you can compute it without the model at all. Second, did the final answer use that chunk correctly? That is an end to end metric, and it usually requires a human or a strong model to judge.

Once you have the test set, the workflow becomes boring in the best way. You change the chunking, you rerun the set, you see if recall went up. You add metadata filtering, you rerun, you see if precision went up. You stop guessing. Every change is a measurement. This is exactly how you would treat a search engine or a ranking system, because that is what this is.

Monitor what the users actually hit

The test set tells you about the past. Production tells you about the future, and production has one metric that matters more than any other: the fraction of queries where the user was not satisfied and there was nothing you could do about it, because the information was not in the corpus.

I call this the coverage problem. A retrieval system can be perfect and still useless if the answer is not in the documents. When you see the same question asked repeatedly, and every answer is a confident paraphrase of "we do not have that information", that is not a retrieval bug. That is a content gap, and the fix is to go get the document, not to tune the index.

The other production signal is the "model overrode the source" family. The retrieved chunks were relevant, but the model answered from memory anyway, or contradicted the source, or answered a question that was not asked. Log the retrieval results alongside the answer. When the answer is wrong, you need to know whether the chunk was wrong or the generation was wrong. If you do not log both, you will spend weeks arguing about which one it is.

When RAG is the wrong tool

Not every question needs retrieval. If your system can answer from a small, stable set of rules, or from a table that fits in a prompt, retrieval is overhead. I have seen teams bolt a vector store onto a problem that a lookup table solved faster and more reliably, because "we are doing AI" felt better than "we are doing a dictionary".

Retrieval earns its keep when the knowledge is large, changes over time, and cannot be encoded in a prompt. If your corpus is five pages, put the five pages in the system prompt and spend your engineering time elsewhere.

The boring version wins

If I have one piece of advice, it is this: the RAG systems that survive contact with production are the ones where the team treated retrieval as a discipline, not a feature. They named their chunks, they kept provenance, they built a test set, they logged retrieval and generation separately, and they accepted that most of their work would be invisible plumbing rather than impressive demos.

The model is a commodity. The corpus is the moat. The sooner your team internalizes that, the sooner the demo stops breaking.