RAG and Retrieval

retrieval-augmented generation (RAG) puts a live, queryable corpus next to the model. In production, that corpus becomes an evidence supply chain: the system selects a small authorized subset and gives it to a model for one response. The user value is not "vector search." It is an answer that can use current, private, or specialized evidence and show where that evidence came from.

Retrieval does not make the answer true. A source may be wrong, an index may be stale, a retriever may miss the decisive passage, a context packer may drop it, or a generator may ignore it. Every stage therefore needs an input contract, an output contract, and a measurement. Retrieval cannot recover a source that was never indexed; generation cannot recover evidence that never reached its context. When the available evidence is insufficient or contradictory, the system needs an explicit way to abstain.

Start with the evidence contract

Lewis et al. introduced RAG in 2020 as a model that combined parametric memory, held in a sequence-to-sequence model, with non-parametric memory in a dense Wikipedia index (Lewis et al. 2020). The term now covers a broader production pattern: retrieve external material at request time, assemble a bounded context, then generate from it. That broader use makes the data contract more important than any particular vector database.

An indexed chunk needs enough identity to survive parsing, re-embedding, permission changes, citations, and deletion. A request needs enough identity to constrain every search path and enough budgets to bound the work. Its freshness bound states how old an acceptable index snapshot may be. A generated claim needs a stable link back to the exact material that supported it.

IndexedChunk {
  tenant, document_id, chunk_id
  text, source_uri, source_version, source_span, content_hash
  title, section_path, language, source_trust
  acl, acl_version, valid_from, valid_until
  parser_version, chunker_version, embedding_model_version, index_version
}

RetrievalRequest {
  request_id, authenticated_principal, tenant
  query, corpus_scope, policy_version, freshness_bound
  candidate_budget, token_budget, latency_budget
}

Candidate {
  chunk_ref, retrieval_channel, raw_rank, raw_score
  fused_score, rerank_score, index_snapshot
}

ClaimSource {
  claim_id, document_id, source_version, chunk_id
  quoted_span, source_uri, retrieval_request_id
}

The fields are not decorative metadata. document_id and chunk_id provide stable document identity across indexes. source_version, source_span, and content_hash identify the exact evidence. The access control list (acl) and its version bind authorization to the same material. The parser, chunker, embedding model, and index versions make a retrieval result reproducible.

This contract separates two systems that are often blurred together. The offline path ingests, parses, chunks, enriches, embeds, and indexes sources. The online path authenticates a request, constructs its authorized candidate set, retrieves, fuses, reranks, packs context, generates, and verifies citations.

Ingestion must preserve meaning and deletion

Parsing is the first retrieval decision. Headers, tables, lists, code blocks, page coordinates, and section ancestry often carry the context needed to interpret a sentence. Optical character recognition can introduce substitutions that look plausible to both an embedding model and a generator. Keep the raw source, parser output, parser version, and source offsets so a retrieved chunk can be checked against the original.

Chunking then chooses the unit of retrieval. A whole report may be too coarse; one sentence may be too isolated. There is no universal chunk length. Start at semantic boundaries such as sections, paragraphs, records, or functions. When a small child chunk is retrieved, parent or neighboring text can be expanded later for interpretation. Overlap can protect boundary facts, but overlapping hits must be deduplicated before context packing or one passage can consume the budget several times.

Choose chunking parameters on representative queries, not intuition alone. Measure whether the complete supporting span appears in at least one indexed unit, how much irrelevant text accompanies it, and how often the unit crosses a permission boundary. A generated contextual prefix can improve retrieval for some corpora, but it is derived data: keep it separate from source text, version the model that produced it, and evaluate it on the intended workload. Anthropic reported a relative reduction in top-20 retrieval failures on its evaluation corpora with this method; that vendor result is not a universal chunking rule (Anthropic 2024).

Updates are part of ingestion. An idempotent write replaces all sparse, dense, and derived records for one source version. A tombstone must block reads as soon as a document is deleted or access is revoked, even if removal from embeddings, graphs, summaries, and caches continues asynchronously. Content and permission metadata must not tear across versions. One response should pin an index snapshot, or record every snapshot it mixed.

Changing the embedding model is a schema migration, not a configuration edit. Store embedding_model_version with every vector, build the new index beside the old one, evaluate both on the same queries, and cut traffic over explicitly. Query vectors and document vectors from incompatible models must never be compared.

Candidate generation uses complementary signals

The first online search stage should be broad and cheap enough to preserve candidate recall. Sparse and dense retrieval provide different signals; neither is the universal fallback for the other. BEIR found BM25 to be a strong zero-shot baseline across heterogeneous retrieval tasks, while late-interaction and reranking systems performed best on average at higher computational cost (Thakur et al. 2021). Domain shift, query type, language, and corpus structure can reverse a ranking observed on one benchmark.

Sparse retrieval preserves lexical evidence

The BM25 lexical baseline, a sparse keyword-scoring method, scores terms through an inverted index. A common BM25 form is

$$\begin{aligned} L_d &= 1-b+b\frac{|d|}{\overline{|d|}}, \\ s_{\mathrm{BM25}}(q,d) &= \sum_{t \in q} \operatorname{IDF}(t) \\ &\quad {}\times \frac{f(t,d)(k_1+1)} {f(t,d)+k_1L_d}. \end{aligned}$$

Here $q$ is the query, $d$ is a document or chunk, and $t$ is a query term. The term frequency $f(t,d)$ counts occurrences of $t$ in $d$; $\operatorname{IDF}(t)$ weights terms that are rare in the corpus; $|d|$ and $\overline{|d|}$ are the document length and average document length; and $L_d$ is their length-normalization factor. The positive constant $k_1$ controls term-frequency saturation, while $b$ between zero and one controls length normalization (Robertson and Zaragoza 2009).

BM25 is lexical, but it is not exact string matching. Tokenization, case folding, stemming, stop-word policy, and field weights all change the result. Identifiers, product codes, error messages, names, and quoted phrases often make sparse retrieval valuable. Test those query cohorts separately from paraphrase and conceptual queries.

Dense retrieval learns a similarity function

A dual-encoder maps the query and each chunk into the same vector space. Its usual inner-product score is

$$s_{\mathrm{dense}}(q,d) = E_Q(q)^{\mathsf T}E_D(d).$$

Here $q$ is the query, $d$ is a chunk, $E_Q$ is the query encoder, $E_D$ is the document encoder, and the superscript $\mathsf T$ denotes vector transpose. If both vectors are normalized to unit length, the inner product is also cosine similarity. Otherwise it is not.

Dense Passage Retrieval showed 9 to 19 percentage-point gains in top-20 passage accuracy over its Lucene BM25 baseline on several open-domain question-answering datasets (Karpukhin et al. 2020). That result established a useful architecture, not a theorem that dense retrieval dominates sparse search. A dense retriever can fail after domain shift, on rare literals, or when its training negatives do not resemble production confusions. Chapter 45 treats that representation and training problem in detail.

Approximate search is an empirical trade

Exact vector search scores every eligible vector. It is a valuable correctness baseline and can be practical for small or strongly filtered collections. At larger scales, approximate nearest neighbor search reduces work by accepting a workload-dependent chance of missing an exact neighbor.

The hierarchical navigable small-world (hierarchical navigable small-world (HNSW)) graph is one common design. It searches a layered proximity graph from sparse upper layers into denser lower layers (Malkov and Yashunin 2020). Parameters governing graph degree, construction effort, and search breadth trade memory, build time, query latency, and empirical neighbor recall. HNSW does not give a general per-query recall bound or a worst-case semantic guarantee. Measure ANN neighbor recall against exact vector search, then measure semantic relevance separately.

A production vector store may use HNSW, a flat index, inverted files, quantization, disk-based graphs, or a combination. It also owns persistence, replication, metadata filtering, deletion, compaction, and consistency. Those properties matter as much as its distance function.

Hybrid retrieval needs stable fusion

hybrid search runs sparse and dense retrieval over the same authorized snapshot. Reciprocal rank fusion (RRF) combines ranks without assuming that scores from the two systems share a scale:

$$s_{\mathrm{RRF}}(d) = \sum_{r \in \mathcal{R}:d \in r} \frac{1}{k_0+\operatorname{rank}_r(d)}.$$

Here $d$ is a document, $\mathcal{R}$ is the set of input rankings, $r$ is one ranking that contains $d$, and rank_r(d) denotes its one-based position in that ranking. The positive constant $k_0$ reduces the influence of a single extreme rank (Cormack et al. 2009). The original study fixed $k_0=60$ after a pilot; that value is a starting point to validate, not a law.

Fusion requires stable document identity. Deduplicate the same chunk returned through multiple channels, preserve every contributing rank, and define a stable tie break. RRF cannot surface an item omitted by every truncated input list. It also ignores score calibration and channel quality; learned or weighted fusion may be better when labeled traffic shows one retriever is consistently stronger.

# RRF uses rank positions, not incomparable raw scores.
def rrf(rankings, k0=60):
    scores = {}
    for ranking in rankings:
        for rank, document_id in enumerate(ranking, start=1):
            scores[document_id] = scores.get(document_id, 0.0) + 1 / (k0 + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

dense = ["A", "B", "C", "D"]
sparse = ["E", "C", "F", "G"]
print(rrf([dense, sparse]))

Reranking cannot repair a missing candidate

A dual-encoder is efficient because the query and chunk are encoded separately. A cross-encoder reads the pair together and can model token-level interactions before assigning one relevance score. This is more expressive but requires a model pass per query-candidate pair. The candidate depth is therefore a measured budget, not a fixed recipe.

The candidate recall ceiling is absolute for the reranker: if the required evidence is absent from its input set, no scoring improvement can restore it. Sweep shortlist depth while measuring candidate recall, nDCG, end-to-end answer quality, latency, and cost. More candidates can help ranking and still hurt the generator if the packed context gains distractors.

ColBERT provides a different point in the design space. It stores per-token document embeddings and uses a late MaxSim interaction with query-token embeddings (Khattab and Zaharia 2020). The interaction preserves finer evidence than one vector per chunk and can support full-corpus retrieval as well as reranking. The price is a larger index and more retrieval-time interaction.

After reranking, context packing selects evidence under a token budget. It should deduplicate overlapping chunks, preserve source order when order matters, expand parent context when needed, and avoid letting one redundant document crowd out independent evidence. Relevance is not sufficiency. A passage can match the question without supporting the requested conclusion.

Every material answer claim should carry a ClaimSource that points to a quoted span and source version. Citation precision asks whether cited sources support their attached claims. Citation recall asks whether claims that need evidence are cited at all. ALCE demonstrated that citation correctness and completeness must be evaluated separately from answer fluency (Gao et al. 2023). A citation makes an answer inspectable; it does not prove that the source is trustworthy or that the model used it faithfully.

Conflicting authoritative passages should remain visible to the answer policy. The system may explain the disagreement, prefer a source under an explicit freshness or authority rule, request clarification, or abstain. Silently picking the highest similarity score turns a ranking artifact into a factual policy.

Authorization constrains every candidate path

Authorization is not a ranking feature and not a cleanup stage. It defines the authorized candidate set before sparse search, dense search, graph traversal, fallback search, fusion, reranking, generation, caching, or logging can expose content. Every retrieval subquery inherits the authenticated principal, tenant, corpus scope, and policy version from its RetrievalRequest.

For every candidate c:
  c.tenant == request.tenant
  authorize(request.authenticated_principal, c.acl, request.policy_version)
  c.valid_from <= request.time < c.valid_until

These predicates must pass before c's text, score, or identifier leaves
candidate generation.

Vector filters illustrate why placement matters. A prefilter constrains the set searched by the approximate index. A postfilter first retrieves from a broader set and intersects later, so selective filters can return too few results or miss eligible documents outside the unfiltered top results. Azure documents the recall, latency, and throughput trade among these modes (Microsoft 2026). For authorization, use a server-constructed prefilter or a physically isolated index. Postfiltering and over-fetching may be ranking experiments, but they are not a security boundary.

Permission evaluation must fail closed. Missing identity, failed group resolution, stale required metadata, or an authorization-service error returns no protected content and no model-only fallback that implies access. Measure retrieval quality against entitlement-scoped gold evidence, especially for users with narrow access. A corpus-wide recall score hides the cohort most likely to receive an empty shortlist.

Revocation needs a short synchronous path and a complete asynchronous path. The synchronous deny decision blocks the source or ACL version immediately. Cleanup then removes affected chunks, embeddings, lexical entries, graph nodes, summaries, prompt fragments, and answer-cache entries. Derived data must inherit an authorization policy at least as restrictive as every contributing source.

A cache key includes the tenant, entitlement or ACL version, index snapshot, query, retrieval configuration, and model configuration. A hit produced for one principal must not become evidence for another. Access-control changes and source tombstones invalidate affected cache entries.

One-shot retrieval is only one policy

Single-shot retrieval works when one query can expose enough evidence. More complex requests may need query decomposition. Independent subqueries can run in parallel; sequential subqueries may depend on entities or constraints discovered earlier. Each subanswer retains its own provenance, and the merger deduplicates evidence before synthesis.

An adaptive loop needs a stop condition, maximum rounds, fan-out limit, token and latency budgets, and an insufficient-evidence terminal state. It must detect when query rewriting has drifted away from the user's request. IRCoT interleaved retrieval with intermediate reasoning on multi-step questions (Trivedi et al. 2023). Self-RAG learned tokens that choose whether to retrieve and assess relevance, support, and response utility (Asai et al. 2024). Adaptive-RAG selected among no retrieval, single-step retrieval, and iterative retrieval by estimated question complexity (Jeong et al. 2024). These are task-specific results, not evidence that a controller improves every query.

Graph-based retrieval addresses a different workload. GraphRAG extracts entities and relationships, builds communities, summarizes them, and uses those summaries for global questions about a corpus (Edge et al. 2024). Its reported gains concern global sensemaking on the authors' evaluation corpora. Graph extraction and summary generation add cost and new errors. Every node, edge, and summary needs links to raw source spans, compatible freshness, and inherited permissions.

Compare any adaptive or graph-based design with a single-shot baseline under a compute-matched budget. Extra queries, a stronger reranker, more context tokens, and a different generator are separate interventions. Without ablation, a gain cannot be assigned to orchestration.

Retrieved text is untrusted data

Retrieved material is untrusted data, not instructions. A trusted storage path does not make every author trusted, and a highly ranked passage does not acquire control authority. Retrieved text can propose facts for the answer, but it cannot grant tool authority, change the system policy, reveal secrets, widen the corpus scope, or select an outbound destination. Those decisions remain in the harness and authorization layer described in Chapter 41 and Chapter 56.

Two attacks need separate names. Corpus poisoning changes what the retriever is likely to return; PoisonedRAG showed that injected passages can steer answers in the evaluated RAG configurations (Zou et al. 2025). Indirect prompt injection puts instructions inside external content so that a model changes behavior when the content is read (Greshake et al. 2023). Relevance filtering alone does not solve either problem.

Preserve publisher, author, source type, signature or collection path, ingestion time, and source trust as provenance. Source trust can affect which evidence is acceptable for a claim, but it cannot widen authorization. Use independent source agreement for high-impact claims, scan and quarantine suspicious ingestion, separate quoted data from control instructions, and keep tool checks outside the model. Evaluate poisoning success rate and attempted exfiltration, not only answer relevance.

Long context changes the frontier, not the contract

Evaluation must localize the failure

An end-to-end answer score cannot tell whether the source was absent, parsing destroyed it, retrieval missed it, packing dropped it, or generation ignored it. Freeze a test contract containing the corpus version, permissions, query set, generator, prompt, retrieval unit, and gold supporting spans. Include answerable and unanswerable queries, then stratify identifier-heavy, paraphrase, multilingual, multi-step, global, fresh-update, and permission-restricted cases.

For query $q$, retrieval recall at depth $k$ can be written as

$$\operatorname{Recall@}k(q) = \frac{|R_q \cap T_k(q)|}{|R_q|}.$$

Here $R_q$ is the set of relevant evidence units for query $q$, $T_k(q)$ is the top $k$ returned units, vertical bars denote set size, and $k$ is the evaluated depth. This definition requires complete relevance judgments; when only one passage is known, report hit rate rather than pretending the labels are complete.

Graded relevance and ordering can be summarized with normalized discounted cumulative gain:

$$\begin{aligned} \operatorname{DCG@}k(q) &= \sum_{i=1}^{k} \frac{2^{\operatorname{rel}_i}-1}{\log_2(i+1)}, \\ \operatorname{nDCG@}k(q) &= \frac{\operatorname{DCG@}k(q)} {\operatorname{IDCG@}k(q)}. \end{aligned}$$

Here $i$ is a one-based rank, $\operatorname{rel}_i$ is the judged relevance of the result at rank $i$, $\operatorname{DCG@}k(q)$ is the observed discounted gain, and $\operatorname{IDCG@}k(q)$ is the maximum possible discounted gain for the same judgments. nDCG@k is useful when several passages have different value; MRR is useful only when the first sufficient result is the intended target.

Measure the pipeline at its boundaries:

• Ingestion: parse success, chunk coverage of gold spans, duplicate rate, source-to-searchable delay, deletion delay, revocation delay, and index age.
• Candidate generation: BM25 and dense recall@k, ANN neighbor recall against exact search, fusion recall, nDCG@k, empty-result rate, and results by query and entitlement cohort.
• Packing: packing recall, relevant-token share, overlap waste, source diversity, and contradictory-evidence retention.
• Generation: supported-answer accuracy, abstention precision and recall, citation precision, citation recall, claim-level faithfulness, and user utility.
• Security: unauthorized-result rate, cross-tenant retrieval, poisoning success rate, prompt-injection success, stale-citation rate, and cache leakage.

Run the same generator with a no-retrieval baseline, an oracle-context baseline, and the actual retrieved context. The oracle-context gap isolates evidence use; the gap between oracle and actual context isolates retrieval and packing. Add BM25-only, dense-only, hybrid, reranked, long-context, and adaptive variants under matched token, dollar, and latency budgets. RAGChecker is one example of a fine-grained framework that separates retrieval and generation diagnostics (Ru et al. 2024).

Use paired per-query comparisons and a bootstrap confidence interval. Repeat stochastic generation, calibrate model judges against blinded human labels, and version judge prompts and models. Then inject operational failures: add a new source, update a fact, delete a source, revoke access, corrupt a parser output, remove supporting evidence, insert plausible distractors, add contradictory evidence, and poison a chunk. The expected response is part of the test contract.

A production retrieval outline

The safe loop is still short, but its arguments carry the system contract:

def answer(request):
    principal = authenticate(request)
    scope = authorize_scope(principal, request.tenant, request.corpus_scope)
    snapshot = pin_index(request.freshness_bound)

    dense = dense_search(request.query, snapshot, prefilter=scope)
    sparse = bm25_search(request.query, snapshot, prefilter=scope)
    candidates = stable_rrf(dense, sparse)
    candidates = verify_versions_and_acl(candidates, principal, snapshot)
    ranked = rerank(request.query, candidates, request.candidate_budget)
    context = pack_with_provenance(ranked, request.token_budget)

    if not evidence_is_sufficient(request.query, context):
        return abstain_with_retrieval_trace(request, snapshot)

    response = generate_from_untrusted_evidence(request.query, context)
    return verify_claim_sources_or_abstain(response, context)

Production behavior also needs a retention contract. Retrieved text may persist in traces, conversations, provider logs, prompt caches, evaluations, or response caches even when it is absent from long-term agent memory. State which copies exist, who may read them, when they expire, and how deletion reaches them.

RAG sits beside the persistent state of Chapter 39. It supplies evidence for a request; it does not become durable memory unless a separate, authorized write accepts it. The next chapter, Chapter 45, opens the dense retrieval box and explains how the representation space is trained, evaluated, compressed, and migrated.

Further reading