Retrieval Failure vs Generation Failure: How to Diagnose Which Layer Is Killing Your RAG System

---

March 2025. I'm sitting with a team that has spent the better part of six weeks improving their AI assistant's prompt. New system message. Firmer instructions. "You MUST use the provided context." Few-shot examples. Each version fixed the last complaint and broke something else. The answers were fluent, confident, and wrong in a fresh way every sprint.

I asked one question: when it answers wrong, what did the retriever actually return?

Nobody knew. Nobody had looked. They had been debugging generation for six weeks without ever confirming that generation was where the failure lived. The right document wasn't in the context window on most of the failing queries — a chunk boundary had sliced the relevant paragraph in half, and neither half alone answered the question. No prompt fixes that. You can't instruct a model to cite a fact it never saw.

This is the most expensive mistake I see in production AI work, and it's not a modeling problem. It's a diagnosis problem. Going AI-native early taught me the failure modes before they were common knowledge, and this is the one that costs teams the most weeks: treating a two-part system as a single black box.

Retrieval and generation are two systems bolted together

A retrieval-augmented pipeline is not one thing. It's a retriever that finds documents and a generator that writes sentences, wired in series. One does search. The other does language. When the output is wrong, exactly one of them is at fault, and the fix for one does nothing for the other.

Here's the pattern that wastes the weeks. An engineer notices a wrong answer. The response reads fine — polished, grammatical, sure of itself. So the instinct is to blame the model. Rewrite the prompt. Add a stronger instruction. Ship it. Test again; wrong in a different way now.

Prompt engineering on top of a broken retrieval layer is decorating a house with no foundation. And teams default to it for a reason worth naming: prompts are a text field you edit in thirty seconds. Re-indexing a vector store is a job. That asymmetry — cheap to edit versus expensive to diagnose against — is the entire reason so many teams fix the wrong layer first, over and over, without noticing the pattern. It's prompt-chasing, and it feels like progress precisely because it's the easiest thing to touch.

The numbers back this up. Across industry post-mortems and the failure-mode literature, retrieval — not generation — is where most production failures originate; analyses put it anywhere from roughly 40% to over 70% depending on how you count, and a widely cited taxonomy catalogs seven distinct failure points, most of them upstream of the model. Yet the prompt is where teams spend their first six weeks. The layer that's cheapest to change is the one you'll wrongly blame first.

"It's wrong" is a symptom, not a diagnosis

"The system is wrong" tells you nothing you can act on. It's an incident ticket that says only the site is slow. Slow where — database, network, client render? You need the same discipline here. Split every failure into two questions, asked strictly in this order:

1. Did the retriever surface the right context?
2. Did the generator use that context correctly?

The first question is answerable without ever running the language model. Take the query, run it through retrieval, look at what comes back. Is the fact that would answer the question sitting in the returned chunks? Yes or no. That's a retrieval question with a retrieval answer, full stop.

The second question only makes sense once you've answered the first. If the correct chunk was in the context window and the model still ignored it, contradicted it, or hallucinated around it — that's a generation failure, and it lives in the prompt, in how instructions compete with retrieved text, in how conflicting chunks get ordered, or in the model's own limits under ambiguity. Interestingly, a large share of what people call "hallucination" is exactly this: the right evidence was retrieved and the model ignored it. That's not a retrieval miss. That's a generation failure wearing a retrieval costume.

Skip the split and you re-ship the same bug three times under three names.

Measuring retrieval independent of generation

To measure retrieval on its own, take the model out of the loop entirely. Build a small, honest set of queries where you already know which chunk contains the answer. Run only retrieval. Check whether the right chunk shows up, and where it ranks.

The core diagnostic is binary: is the right chunk in the top-K results? Yes or no. That single check — Hit@k in the literature — isolates retrieval quality from generation quality and bypasses any subjective read of the model's prose. Most production systems should be clearing Hit@5 above 80% before anyone touches a prompt. If you want more resolution, context precision and context recall tell you whether what you retrieved was relevant and whether what was needed got retrieved; MRR and nDCG tell you whether the right chunk sits near the top or is buried. Frameworks like RAGAS exist to automate exactly this. But the yes/no top-K check is the one you cannot skip.

Sounds obvious. Most teams never do it. They eyeball the final answer and reason backward, which is slower and noisier — a wrong final answer could be a retrieval miss, a ranking demotion, or a generation error, three causes behind one symptom. Testing retrieval in isolation collapses that ambiguity immediately. Either the right chunk is in the top results, or it isn't.

And the answer tells you which fix:

• The correct chunk isn't there at all. That's upstream of ranking — chunking, coverage, or embeddings. Something never made it into a retrievable form.
• The correct chunk is there but buried below four irrelevant ones. That's re-ranking, not chunking. The embedding step found it; the ranking step demoted it.

Conflate these and you might rebuild your entire chunking strategy to fix what was a ranking threshold set too loosely.

This work is tedious. Nobody wants to hand-label test queries against known-correct chunks, and it won't demo in a sprint review the way a shiny new prompt does. It's also the only way to know — with any confidence — that retrieval works, as opposed to appearing to work because the last five queries you tried by hand happened to be easy ones. This is where the chunking, re-ranking, and hybrid-search deep dive earns its keep: it's the fix menu once you know retrieval is the problem.

Measuring generation independent of retrieval

Once you trust retrieval — or at least know its failure rate — isolate generation the same way. Feed the model the correct context directly, bypassing the retriever. Force the right chunk into the prompt every time, and check whether it produces the right answer.

If it still gets it wrong with the correct context handed to it directly, no amount of re-ranking or chunk-size tuning saves you. That's a prompt problem, a context-formatting problem, or a model-capability problem — instructions competing with retrieved text, the model unsure which source to trust, phrasing in the system message that reads as ambiguous in a way nobody noticed until now.

This is where teams over-index on model changes without asking the prior question. Swapping models, adding elaborate reasoning instructions, nudging temperature — these are generation-layer fixes. Worth doing, but only after confirming generation is where the failure sits. Applying them to a retrieval problem is debugging a client-side bug by upgrading the database. It might coincidentally help. It isn't solving anything.

Building the query set for both of these — retrieval-only and generation-only — is the same discipline as building an eval harness for any non-deterministic system. If you don't have one yet, start there: an eval set is the regression net that turns "it feels better" into "Hit@5 went from 71% to 88%."

Picking the fix: chunking, re-ranking, or the prompt

Once you know which layer is failing, the fix stops being a guess.

If the correct information never makes it into retrieval at all, look at chunking first. Are chunks too large, burying the relevant sentence in noise? Too small, stripping the surrounding context that made it identifiable? Is a boundary cutting the answer in half? This is chunking theater — tuning chunk size without checking whether the split respects the actual structure of the document. A table. A numbered list. A clause that only means anything with its heading still attached. In my experience this is the single most common — and most neglected — root cause. Teams give chunking five minutes and a fixed character count, then spend six weeks on the prompt.

If the correct chunk is retrieved but ranked too low to make the context window, that's re-ranking. The embedding step found it; the ranking step buried it. The fix is adjusting the re-ranker, widening how many candidates it considers, or reconsidering the similarity metric — not touching chunking at all.

If the right chunk lands in front of the model and it still answers wrong, that's the prompt. Or the model. Or how context is structured inside the prompt. This is the layer teams reach for first and should reach for last, because it's the cheapest to edit and the easiest one to convince yourself you've fixed something when you haven't.

One caveat, because the discipline has a cost too. Reserve this rigor for systems where retrieval quality carries real consequences — multi-tenant data boundaries, factual answers where a wrong citation is worse than none, anything in front of real traffic at real scale. Below that bar, ship the guess and see what breaks; it's cheap to be wrong there. Above it, the ordering — retrieval first, then generation — isn't optional. It's the sequence that tells you which layer to actually fix.

What to do Monday morning

1. Grab your last ten wrong answers. For each one, run the query through retrieval alone and look at what came back. Don't run the model. Just check: was the answer-bearing chunk in the top results, yes or no?
2. Tally the split. Count how many were "chunk wasn't there," "chunk was there but ranked low," and "chunk was there in the top results." That tally is your diagnosis. Most teams are shocked how few land in the third bucket.
3. For the ones where the chunk was present, feed it to the model directly and see if the answer comes out right. Now you know your true generation failure rate, uncontaminated by retrieval.
4. Fix the biggest bucket first, not the easiest one. If most failures are "chunk wasn't there," every hour on the prompt is wasted. Fix chunking.
5. Freeze those queries into an eval set. You just built the start of a regression net. Run it before and after every change so "better" becomes a number.

Key takeaways

• A wrong answer is one of two failures — retrieval or generation — never a vague "the system is wrong." Diagnosis precedes fixing.
• Ask two questions in order: did retrieval surface the right context, then did generation use it. The first is answerable without ever running the model.
• The binary top-K check (is the right chunk in the results?) isolates retrieval from generation faster than any end-to-end eyeballing.
• Chunk absent → chunking, coverage, or embeddings. Chunk present but low → re-ranking. Chunk present and used wrong → the prompt or the model.
• The prompt is cheapest to edit, which is exactly why it's the layer teams wrongly blame first. Reach for it last.

Your next step

Pick one failing query today. Run retrieval on it with the model switched off, and answer one question: was the right chunk in the top results? That single yes-or-no tells you which half of your system to stop blaming. Everything else follows from it.

Frequently asked questions

What's the difference between a retrieval failure and a generation failure in RAG?

A retrieval failure means the system never surfaced the right document or chunk — the model worked with incomplete or wrong material. A generation failure means the right chunk was in the context window and the model still got it wrong. They look identical from the outside. Only isolated testing tells them apart: run retrieval alone to check the first, then feed the correct chunk to the model directly to check the second.

What percentage of RAG failures are retrieval versus generation?

Most originate in retrieval. Industry post-mortems and the failure-mode literature put retrieval-side failures anywhere from roughly 40% to over 70% of the total, depending on how they're classified — and a widely cited taxonomy lists seven distinct failure points, most of them upstream of the model. The practical takeaway isn't the exact number; it's that the prompt, where teams spend their first effort, is statistically the least likely culprit.

How do you test retrieval quality without running the language model?

Build a small set of queries where you already know which chunk holds the answer. Run only the retrieval step and check whether that chunk comes back and where it ranks — this is Hit@k, and most production systems should clear Hit@5 above 80%. No generation involved. The check is binary: the right chunk is in the top-K results, or it isn't.

How do you test generation quality independent of retrieval?

Skip the retriever. Feed the model the correct chunk directly, every time, and see if it produces the right answer. If it still fails with the right context handed to it, the problem is generation — the prompt, how context is formatted, or the model itself — and no amount of chunking or re-ranking work will touch it.

If retrieval finds the right chunk but ranks it too low, is that a chunking problem or a re-ranking problem?

Re-ranking. The chunk exists and got embedded correctly — the ranking step buried it below less relevant results. Rebuilding your chunking strategy won't fix a ranking threshold set too loosely. Adjust the re-ranker, over-fetch more candidates, or reconsider the similarity metric instead.

Why doesn't prompt engineering fix most RAG failures?

Because most failures happen before the prompt ever runs — the retriever never surfaced the right information. A better prompt can't make the model cite a fact it never saw. Prompt changes only matter once you've confirmed, by feeding the model correct context directly, that generation is actually where the failure sits.

When is this level of diagnostic rigor actually worth the effort?

When retrieval quality has real consequences — multi-tenant data boundaries, factual answers where a wrong citation is worse than silence, anything in front of real production traffic. Below that bar, guessing is cheap enough to live with. Above it, skipping the diagnosis just means re-shipping the same bug under a different name.