RAG (Retrieval-Augmented Generation)
Also known as: retrieval augmented generation, RAG
RAG is the pattern of retrieving relevant documents at query time and placing them in the model's context so the answer is grounded in your data rather than in the model's parameters. It substitutes retrieval quality for training, and its failure modes are mostly retrieval failures, not generation failures.
Last reviewed · Part of the Architecture Glossary
In practice
The pipeline, with the decision at each stage:
- Chunk. Size and boundary matter more than the embedding model. Semantic or structural chunking (by heading, by function) beats fixed 512-token windows on almost every corpus.
- Embed and index. See vector index.
- Retrieve. Hybrid — dense vectors plus BM25 — outperforms either alone, because keyword search still wins on identifiers, error codes and rare product names that embeddings smear.
- Rerank. A cross-encoder over the top 50 candidates, keeping the top 5. This is usually the single largest accuracy gain in the whole pipeline and the stage most often skipped.
- Generate, with the instruction to answer only from the provided context and to say so when it is insufficient.
Measure the stages separately. Retrieval: recall@k, MRR. Generation: faithfulness against the retrieved context. A single end-to-end accuracy number cannot tell you which half to fix.
When it matters
Any assistant answering over private, current or large-corpus data — documentation, tickets, contracts, code.
Common mistake
Debugging RAG by editing the prompt. If the right chunk was never retrieved, no prompt fixes it. Log the retrieved chunks for every failed answer; most "hallucination" bug reports turn out to be recall bugs.
See also
- EmbeddingAn embedding is a dense vector representation of text, an image or other content, positioned so that semantically similar items sit close together under a distance measure such as cosine similarity.
- Vector Index (ANN)A vector index accelerates nearest-neighbour search over high-dimensional embeddings by trading exactness for speed.
- Context WindowThe context window is the maximum number of tokens a model can attend to in one request — system prompt, conversation history, retrieved documents, tool definitions, tool results and the response combined.
- HallucinationA hallucination is model output that is fluent and confident but not supported by the input or by fact — an invented citation, a non-existent API method, a fabricated figure.