RAG and Retrieval

A model's weights are a lossy, frozen snapshot of the corpus it was trained on. retrieval-augmented generation (RAG) puts a live, queryable corpus next to the model and feeds the right passages into the prompt at answer time. RAG is the funnel that turns a question into a few hundred tokens of evidence: chunk and index the corpus, embed the query, retrieve and fuse candidates, rerank, and generate against the selected passages. Each stage prevents a specific failure and charges a specific cost. The field's later systems attack the funnel's weak points, while two forces decide how much of the funnel survives: an adversary who hides in the corpus and a context window that keeps growing.

Knowledge inside the weights, or outside them

A frontier model knows what was in its pre-training mixture, up to the day the mixture was frozen, blurred by the lossy compression of Chapter 5. It does not know your private documents, it does not know what happened after the cutoff, and when asked about a fact it half-remembers it will produce a fluent, confident, wrong answer. Fine-tuning the knowledge in is expensive, slow to update, and still lossy. The cleaner move is to keep the knowledge outside the weights and fetch it on demand.

That reframes the problem as information retrieval married to generation. Given a question and a corpus of millions of passages, find the small set of passages that actually answer the question, place them in the context window, and let the model read and cite them. The hard parts are all in the retrieval. The corpus is large, so search must be sub-linear. The question and the answer rarely share exact words, which leaves lexical match alone brittle. And the context window is finite and expensive: you can pass only a handful of passages, so they had better be the right ones. Retrieval is the act of turning a question into the few hundred tokens of evidence that change the answer. Lewis et al. named retrieval-augmented generation and framed it as combining parametric memory, the model's weights, with non-parametric memory, a retrievable index, training the retriever and generator together on knowledge-intensive tasks (Lewis et al. 2020).

Down the funnel, one stage at a time

The naive RAG pipeline that froze out of that idea has four stages: chunk and index the corpus offline, embed the query online, retrieve the nearest chunks, and generate an answer conditioned on them. Each stage exists to answer a specific failure of the stage that would otherwise replace it, and each carries its own dial with a knee. We walk them in order.

Chunking exists because documents are too long to embed or retrieve whole. A single vector cannot represent a hundred-page report at the granularity a question needs, and you cannot paste the whole report into the prompt. So the corpus is split into chunks, typically a few hundred tokens, and each chunk is indexed independently. The chunk size is a direct trade with no universal answer: chunks too large dilute the relevant sentence among irrelevant ones and waste context budget; chunks too small sever the sentence from the context that disambiguates it. The knee depends on document structure and question type, which is why overlapping windows and structure-aware splitting (by section, by paragraph) exist to soften the boundary problem. A third softener augments the chunk itself: contextual retrieval prepends an LLM-generated sentence situating each chunk in its document before both the embedding and the keyword index are built, which Anthropic reports cuts top-20 retrieval failures by 49% (Anthropic 2024).

Dense retrieval answers the brittleness of lexical match. A dual-encoder embeds the query and each chunk into the same vector space, trained so that a question and its answer land near each other even when they share no words. How a piece of text becomes that vector is the subject of Chapter 45. Karpukhin et al., in Dense Passage Retrieval, showed that a dual-encoder trained on a few thousand question-passage pairs beats a strong BM25 lexical baseline, a sparse keyword-scoring method, by a wide margin on top-20 retrieval accuracy across open-domain QA (Karpukhin et al. 2020). Relevance becomes a dot product or cosine similarity in embedding space, a closeness score between two vectors:

$$\operatorname{sim}(q, c) = \mathbf{E}_q(q) \cdot \mathbf{E}_c(c)$$

where $\mathbf{E}_q$ and $\mathbf{E}_c$ are the query and chunk encoders. The encoders can be tied or separate, and the chunk embeddings are computed once, offline, then frozen into an index. The dial here is dense against sparse: dense retrieval generalizes across paraphrase and fails on exact tokens, which the next stage corrects.

The vector store is what makes dense search affordable at scale. With millions of chunk vectors, computing the dot product against every one for every query is too slow. Approximate nearest neighbor indexes trade a small, bounded recall loss for orders of magnitude in speed. The dominant structure is the hierarchical navigable small-world (HNSW) graph of Malkov and Yashunin, which builds a multi-layer proximity graph and searches it greedily from a sparse top layer down, giving logarithmic-scale query time (Malkov and Yashunin 2018). A vector store is HNSW (or a quantized variant) plus metadata filtering plus the plumbing to keep the index in sync with the corpus. The dial is recall against latency: exact nearest neighbor is correct and too slow, and the index parameters are a recall-latency knob, not a fixed setting.

hybrid search answers the blind spot of dense retrieval. Embeddings generalize across paraphrase but lose exact tokens: a part number, a rare name, a code identifier, a literal string that the dense model has smoothed away. Sparse lexical retrieval, BM25 over an inverted index, is exact on those terms and complementary. Hybrid search runs both and fuses the rankings, commonly with reciprocal rank fusion, so that a chunk ranked highly by either method surfaces. It pays two indexes and a fusion step to get both halves, and that cost is usually worth it.

Try giving a chunk a mediocre rank in both lists and watch reciprocal rank fusion still float it above items that ranked first in only one list.

# Reciprocal rank fusion: score = sum over lists of 1/(k + rank).
def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

dense  = ["A", "B", "C", "D"]   # what embeddings rank highly
sparse = ["E", "C", "F", "A"]   # what BM25 ranks highly
fused = rrf([dense, sparse])
print("dense :", dense)
print("sparse:", sparse)
print("fused :", fused)
print("C wins: ranked 3rd and 2nd, never 1st, but agreed on by both.")

Reranking addresses a different gap, the one between cheap retrieval and accurate scoring. The dual-encoder is fast because the query and chunk never meet until the final dot product, which is also why it is imprecise: it cannot model fine-grained interaction between the query's words and the chunk's. A cross-encoder reranker feeds the query and chunk together through a transformer and scores their joint representation, which is far more accurate and far too slow to run over the whole corpus. So the pipeline retrieves a few hundred candidates cheaply, then reranks the top candidates expensively, and the length of that shortlist is the dial: running the cross-encoder only on a retrieved shortlist buys most of the accuracy at a fraction of the cost. ColBERT sits between the two: its late-interaction architecture keeps per-token embeddings and does a cheap MaxSim interaction, recovering much of the cross-encoder's accuracy at retrieval-time cost (Khattab and Zaharia 2020).

Drag the shortlist depth and watch answer accuracy climb fast at first, then flatten: the first dozen reranked candidates carry most of the gain, while a longer shortlist costs cross-encoder compute for diminishing returns.

The cost and accuracy of these three architectures follow directly from one design choice: where in the network the query and the chunk first meet, shown in Figure 44.4.

Stacked end to end, the four stages form a funnel, shown in Figure 44.5.

Each stage is cheaper-and-broader or costlier-and-sharper than the next, and the order exists so that the expensive stages only ever see a short list. This is the workhorse design, and its strength is exactly its rigidity: embed once, retrieve once, generate once, with every dial set ahead of time.

Three weaknesses, three lines of attack

That rigidity is also where the funnel breaks. The naive pipeline decouples its stages and runs each exactly once, which gives it three structural weaknesses, and most of the field's evolution is one line of work per weakness, Figure 44.6.

The first weakness is local-only reasoning. Naive RAG retrieves the chunks most similar to the query, which works for questions whose answer sits in a few passages and fails for questions whose answer is spread across the whole corpus. Asked "what are the main themes in these documents?", similarity search has nothing to grab, because no single chunk is about the themes. GraphRAG reframes this as query-focused summarization: it uses a model to extract an entity-and-relationship graph from the corpus, runs Leiden community detection to partition the graph hierarchically, and pre-summarizes each community. A global question is then answered by map-reduce over community summaries rather than over raw chunks, which the authors show improves the comprehensiveness and diversity of answers on sensemaking questions over corpora in the million-token range (Edge et al. 2024). LightRAG keeps the graph idea but cuts the cost: a dual-level retrieval that combines low-level entity retrieval with high-level concept retrieval over a graph index, with incremental updates so a new document does not force a full re-index (Guo et al. 2024).

The second weakness is the static, single-shot control flow. Naive RAG always retrieves, retrieves exactly once, and never checks whether what it retrieved was any good. The fix is to put a controller in the loop. Self-RAG trains the model to emit reflection tokens that decide whether to retrieve at all and then critique each retrieved passage for relevance and support before using it (Asai et al. 2023). CRAG adds a lightweight evaluator between retrieval and generation that scores retrieved chunks and, when they fall short, triggers a corrective web search instead of generating from weak evidence (Yan et al. 2024). These are the bridge from naive to agentic.

The third move generalizes the second. Singh et al., surveying agentic RAG, describe the endpoint: the retrieval loop becomes an agent that plans, decomposes a question into sub-queries, chooses which tools and indexes to query, reflects on intermediate results, and iterates until it has enough evidence (Singh et al. 2025). Retrieval stops being a fixed funnel and becomes a policy the agent runs, drawing on the agent architectures of Chapter 38 and, when multiple specialized retrievers are involved, the coordination of Chapter 43. The same passages, reordered and re-queried, get used very differently. The dial across this whole axis is naive against agentic: a single-shot pipeline is cheap, predictable, and low-latency; an agentic loop answers harder questions but multiplies token cost and latency and adds failure modes of its own. Most production traffic does not need the agent; the minority that does benefits enormously.

Agentic control also exposes an assumption the funnel never questioned: that retrieval needs an index at all. Coding agents largely dropped it. When the corpus is a filesystem the agent can navigate, it searches live state with grep, glob, and file reads inside the same plan-act-observe loop, and a study of agent harnesses finds grep-style search generally beating vector retrieval in that setting, though the margin depends on the harness and its tool-calling style (Sen et al. 2026). The trade is stark on both sides: no chunking, no embedding drift, no re-index lag, and always-fresh state, paid for in tool-call round trips and tokens on every query. The pattern works where the corpus has native structure to navigate, a repository or an API; at web scale nothing is greppable, and the funnel remains the only way in.

Retrieved text is untrusted input

Adding retrieval moves the security boundary. Once untrusted text from a corpus flows into the prompt, that text can carry instructions. SafeRAG benchmarks this directly, classifying attacks into silver noise, inter-context conflict, soft advertisement, and white denial-of-service, and finds that across many RAG components even simple injected passages bypass retrievers and filters and degrade answer quality (Liang et al. 2025). Retrieved content is untrusted input and must be treated as such, which connects directly to Chapter 56. This is a weakness the funnel cannot tune away: no chunk size or rerank depth makes a planted instruction safe, because the problem is trust, not ranking.

Will long context make retrieval obsolete?

The second force pressing on retrieval is friendlier in appearance and more fundamental: the context window keeps growing, and a large enough window might make the whole funnel unnecessary.

The debate has a clean resolution once you ask who pays for the window, and the answer comes from a layer well below retrieval.

Making it real

A minimal naive RAG loop is short, and the brevity is the point: the sophistication lives in the index, the embeddings, and the reranker, not the control flow.

def answer(query, k=5):
    q = embed(query)
    dense = ann_index.search(q, n=100)         # HNSW recall
    sparse = bm25.search(query, n=100)          # exact terms
    fused = reciprocal_rank_fusion(dense, sparse)
    top = cross_encoder.rerank(query, fused)[:k]
    return generate(prompt=build_context(query, top))

Three implementation realities decide whether this works in practice. The first is index freshness: the offline chunk-and-embed step must run incrementally as the corpus changes, or retrieval silently serves stale evidence. The second is evaluation, which is harder than it looks, because a RAG system can fail at retrieval (the right chunk was never fetched) or at generation (the right chunk was fetched and ignored or misread), and these need separate measurement. Retrieval quality is measured with ranking metrics like recall at k and nDCG; answer quality with faithfulness to the retrieved evidence and answer relevance, typically judged by a model as in Chapter 50, with the caveats about agentic and trajectory evaluation in Chapter 47. The third is the provenance contract: because every claim should trace to a retrieved chunk, the pipeline must carry citations through to the answer, which is both the original selling point of RAG in Lewis et al. (Lewis et al. 2020) and the main defense against the confident hallucination that motivated it.

RAG sits beside, not inside, the agent's working memory. The retrieved context is ephemeral, assembled per request and discarded, which distinguishes it from the persistent state of Chapter 39. Retrieval is how an agent reaches knowledge it does not hold; memory is how it keeps what it has learned. The two compose, and the strongest systems use both.

Further reading