Retrieval and Document Intelligence

Retrieval is not a database feature bolted onto a model. It is an evidence service with two jobs: publish trustworthy representations of source material, then return only the evidence a caller is allowed to see. The theory in Chapter 44 explains retrieval-augmented generation (RAG), while Chapter 46 explains how selected evidence becomes model input; this chapter turns both into an operable system. The result of the work is a retrieval release: a versioned corpus, a tested query path, and the evidence needed to deploy or roll it back (Lewis et al. 2020).

Start with a contract, not a product shortlist. Record the corpus boundary, the query classes the system must answer, the success evidence required for release, and the authority under which each source may be used. Add freshness and deletion service levels, latency and cost budgets, and a rollback window. Without these facts, a fast index or an impressive benchmark score has no operational meaning.

Figure 86.1 separates offline publication from the authorized online path. Evaluation observes both; it is not a final check attached only to the generated answer.

Preserve evidence lineage

A document is not a string. A PDF can contain a digital text layer, page images, annotations, fonts, form fields, tags, and hidden content; these views can disagree (International Organization for Standardization 2020). Store the original source object and its content digest before deriving anything. Give every changed byte sequence a new source revision. A renderer then creates a rendition; a parser produces a document element; a segmenter produces a chunk ID and exact source span; an embedding model produces a representation in one declared space; and an index snapshot publishes it. This provenance chain must remain resolvable from a returned citation (World Wide Web Consortium 2013).

A useful lineage record includes the source identifier, revision and digest; media type and acquisition time; tenant, access-control policy, rights, and retention state; renderer, parser, schema, normalizer, and segmenter revisions; page coordinates and reading order; raw and normalized text; parent element and heading path; and every published index generation. Keep raw evidence separate from normalized search text so that normalization can be changed without rewriting history.

Ingestion is a publication protocol

Treat ingestion like a release pipeline:

1. Acquire and authorize. Resolve the source identity, bytes, rights, tenant, retention rule, and current access-control state.
2. Inspect. Verify media type independently of the filename; scan for malware; enforce byte, page, pixel, archive, and execution limits. Reject or quarantine malformed, encrypted, active, or suspicious input.
3. Normalize and render. Preserve the original, then create deterministic renditions with explicit page boxes, rotation, coordinate origin, and units.
4. Parse. Read the digital text layer where reliable; otherwise apply OCR, layout analysis, table and equation recognition, and reading-order recovery.
5. Extract and validate. Produce document elements and structured extraction with evidence selectors. Retain uncertainty rather than inventing values.
6. Segment and embed. Create retrieval units within authorization and retention boundaries; reject silent truncation at model input limits.
7. Stage, reconcile, and publish. Check counts, hashes, omissions, tombstones, and coverage. Atomically publish a complete generation or keep the previous one. Partial output is never quietly promoted.

Retries must be idempotent. A worker crash on page 17, a duplicate delivery, or the same URI returning changed bytes must produce a reconciled revision, not a half-old, half-new index.

Measure document understanding

optical character recognition (OCR), optical character recognition, recognizes text from pixels. vision-language model (VLM), a vision-language model, reads page images and text together. Neither term includes every document task: detecting layout, restoring reading order, reconstructing a table, recognizing an equation, and mapping coordinates are separate operations. Born-digital text extraction is separate again. A toolkit such as Docling can compose several of these operations, while an end-to-end model may emit a structured representation directly (Auer et al. 2024).

There is no universal parser winner. Build a held-out corpus stratified by the documents that can fail in materially different ways: born-digital and scanned pages, multiple scripts, right-to-left text, rotations, low resolution, multiple columns, forms, tables, equations, handwriting, redactions, hidden text, and malformed files. OmniDocBench is useful background, but a public average cannot replace this document stratum evaluation (Ouyang et al. 2025).

For transcription, report character error rate (CER):

$$\mathrm{CER}=\frac{S_c+D_c+I_c}{N_c},$$

where $S_c$, $D_c$, and $I_c$ count character substitutions, deletions, and insertions, and $N_c$ is the number of characters in the reference. Do not use CER as a proxy for every other failure. Measure element detection precision/recall, reading-order error, table structure similarity, formula recognition, field-level precision/recall, and provenance-locator accuracy separately. Review low-confidence, high-impact cases with human review and record the review rate rather than hiding it in a single score.

Structured extraction has two kinds of validity

JSON Schema can prove that output has the permitted shape and types. It cannot prove semantic correctness. A schema-valid invoice with the wrong total is still wrong. For each field, preserve an evidence selector and an explicit unknown state; let the extractor abstain when evidence is missing or ambiguous. Apply domain postconditions such as subtotal arithmetic, date ranges, identifier checksums, and cross-field consistency. Automatic retries must not turn uncertainty into fabricated certainty.

Visual retrieval is an evaluated alternative

Text retrieval is not the only design. ColPali-style visual retrieval embeds a page image and uses late interaction to match it against a query (Faysse et al. 2025). It can retain layout signals lost by text conversion and can work well for visually rich or multilingual documents. It also changes the cost, storage, accessibility, explainability, and generator requirements.

Compare text retrieval, visual retrieval, and a multimodal combination on the same labeled queries. Even a parse-light path still needs stable page identity, authorization, provenance, accessible text or an equivalent accommodation, and structured extraction when a downstream system needs fields rather than pixels.

Segment for retrieval and citation

The retrieval unit is what ranking returns; the citation unit is the smallest source region a reader can verify. They need not be identical. A retriever may score a proposition or passage, then expand to its parent element or section for context. Every unit should retain a stable chunk ID, parent element, source revision and source span, heading path, language, page and coordinates, content digest, policy, and valid-time interval.

Chunk size is not a universal constant. Compare structure-aware elements, fixed windows, sentences, propositions, and parent-child expansion using the same query set. Version the boundary and overlap policy. Overlap can protect boundary evidence, but it also increases index size and creates duplicate results; context assembly must detect duplicates without discarding useful corroboration. A chunk must never cross an authorization or retention boundary.

Authorize the online query

The query contract contains the query, a verified principal and tenant, validated metadata filters, an as-of time, result and context budgets, ranking policy, and resolved index generation. Treat access control as eligibility, not as a relevance feature. The system must authorize before candidate generation, using current resource state; post-filtering a global top-k can leak data and underfill the authorized result.

For query $q$, subject $s$, evaluation time $t$, and immutable generation $g$, define the authorized universe

$$\begin{gathered} U(q,s,t,g)=\\ \{c\in I_g\mid\\ P(s,\mathrm{read},c,t)=\mathrm{allow},\\ c.\mathrm{valid\_from}\le t<c.\mathrm{valid\_to},\\ \neg c.\mathrm{tombstone}\}. \end{gathered}$$

Here $I_g$ is index generation $g$, $c$ is a candidate, and $P$ is the policy decision for subject $s$ at time $t$. Retrieval returns top-k only from $U$. Missing or invalid policy metadata fails closed. The wrong tenant must never appear in candidates, reranker input, debug output, traces, or caches; this is the retrieval analogue of Chapter 56.

Choose representations by measured behavior

Retrieval families make different trade-offs:

• Sparse retrieval such as BM25 scores analyzed terms. It is a strong lexical baseline for names, identifiers, and rare terms, but its analyzer, language rules, field weights, and parameters are part of the index version.
• Dense retrieval uses a dual encoder: document representations can precompute offline, while the query is encoded online. Learned similarity can connect paraphrases, but it can also blur decisive lexical distinctions (Karpukhin et al. 2020).
• Late interaction keeps token-level document representations and combines their matches at query time. ColBERT uses this as a first-stage retriever, not merely as a reranker (Khattab and Zaharia 2020).
• A cross-encoder scores each query-document pair jointly. It is often more expensive and is normally applied to a bounded candidate union. It cannot recover evidence that candidate generation omitted.

These families exchange places across datasets. BEIR and MTEB are useful for method comparison and model screening, but local query strata decide a release (Thakur et al. 2021; Muennighoff et al. 2023). Begin with a lexical baseline, then measure each additional representation and reranker as an ablation.

Freeze the embedding compatibility contract

An embedding is a model-specific representation whose similarity reflects its training objective. Record the model and tokenizer digest, embedding dimension, normalization, distance metric, query prefix, document prefix, truncation rule, and input limit. Cosine similarity, inner product, and Euclidean distance are not interchangeable unless their normalization assumptions make them so.

Query and document vectors must name the same embedding-space revision. Reject dimension, prefix, normalization, or space mismatches. A model change normally requires a new generation and re-embed migration; never infer compatibility from equal dimensions or similar model names. Route query encoding and index selection atomically, even if models are served through the gateway described in Chapter 82.

Fuse ranks without pretending scores agree

Sparse, dense, and visual scores are usually not calibrated to one another. Reciprocal rank fusion (RRF) combines ranks instead (Cormack et al. 2009). For the set of ranked lists $\mathcal{L}$, define

$$s_{\mathrm{rrf}}(d)= \sum_{L_j\in\mathcal{L}:d\in L_j}\frac{w_j}{k_0+r_j(d)},$$

where $d$ is a candidate, $L_j$ is ranked list $j$, $r_j(d)$ is its one-based rank in that list, $w_j\ge0$ is the list weight, and $k_0>0$ is the fusion constant. A candidate not returned by a list contributes zero. RRF does not calibrate relevance scores or use their magnitude. Candidate depth, weights, the fusion constant, missing-item behavior, and deterministic tie-breaking are ranking-policy settings to tune on validation data and freeze before holdout evaluation.

This small implementation makes those semantics testable:

def reciprocal_rank_fusion(ranked_lists, k0, weights=None):
    """Return (document, score) pairs in deterministic descending order."""
    if k0 <= 0:
        raise ValueError("k0 must be positive")
    weights = weights or [1.0] * len(ranked_lists)
    if len(weights) != len(ranked_lists):
        raise ValueError("one weight is required for each ranked list")

    scores = {}
    for weight, ranked in zip(weights, ranked_lists):
        if weight < 0:
            raise ValueError("weights must be non-negative")
        for rank, document_id in enumerate(dict.fromkeys(ranked), start=1):
            scores[document_id] = scores.get(document_id, 0.0) + weight / (k0 + rank)
    return sorted(scores.items(), key=lambda item: (-item[1], item[0]))

Reranking and context selection remain separate decisions. Measure candidate depth, rerank depth, final top-k, and the context budget independently. Enforce the latency budget and input limit of the reranker. Diversity controls such as maximum marginal relevance can reduce redundant context, but they trade relevance against diversity and do not guarantee deduplication.

Publish an index generation

A vector index supplies exact or approximate neighbor search. A production store must additionally meet the index contract for durability, metadata, filters, consistency, backup, recovery, tenant isolation, and migration. Record:

• corpus watermark and source coverage;
• metadata schema, policy fields, and filter selectivity distribution;
• analyzer, embedding space, distance, and ranking-policy revisions;
• vector count, dimension, hashes, tombstones, and duplicate policy;
• index algorithm and parameters, build hardware, duration, and recovery time;
• exact-search baseline, ANN recall, latency percentiles, concurrency, and cost;
• supported update, delete, backup, restore, and rollback behavior.

HNSW is a graph-based approximate-neighbor design whose memory, build, recall, and latency trade-offs depend on the workload (Malkov and Yashunin 2020). DiskANN-style designs use SSD-aware graph search to change that resource balance (Subramanya et al. 2019). Neither name chooses a store for you. Compare exact scan, HNSW, DiskANN, IVF or compression variants under realistic filters, updates, concurrency, cache state, and tenant skew. Report ANN recall against the exact eligible top-k at each important filter selectivity.

A release manifest can be small and still prevent accidental mixing:

generation: corpus-2026-08-07-03
source_watermark: "2026-08-07T08:00:00Z"
parser: doc-parser@sha256:...
segmenter: headings-v4
embedding_space: embed-model-v7-d1024-cosine
ranking_policy: hybrid-rrf-rerank-v5
policy_schema: acl-v3
deletion_watermark: "2026-08-07T08:05:00Z"

Build a new immutable generation as a shadow index. Backfill it, reconcile source counts, digests and tombstones, and catch up changes to an explicit watermark. Run shadow queries, then dual read old and new generations. Compare quality, authorization, freshness, latency, and cost by query class. Use an atomic routing change that moves both the query encoder and the index; retain the last-known-good generation through the rollback window. Never search an old index with a vector from an incompatible new embedding space.

Freshness and deletion cover every copy

Define the freshness SLA from a committed source change to an eligible result, and the deletion SLA from revocation to absence. A deletion creates a tombstone that propagates to the vector index, lexical index, reranker and answer cache, published snapshot, replicas, and every in-flight migration generation. Logical suppression must happen promptly; physical erasure follows the declared retention process. Cache keys include tenant, policy version, generation, and as-of time. A stale result is explicit and never silently broadened as fallback.

Test insert, update, ACL change, source deletion, interrupted backfill, duplicate replay, partial index failure, corrupt snapshot, and rollback. Prove that a deleted document cannot reappear from a cache, old snapshot, or shadow index.

Return evidence, not anonymous text

The retrieval response is an evidence bundle. Each item carries an immutable evidence ID; source object, source revision, digest, span or page coordinates; index and ranking revisions; stage-specific ranks; applied policy and time; and truncation, underfill, timeout, stale-result, and fallback flags. Preserve unlike scores as unlike scores rather than presenting them as one calibrated probability.

Bind every material answer claim to one or more retrieved evidence IDs. Measure citation precision (cited evidence supports its claim), citation recall (supported claims that have a citation), locator validity, entailment or contradiction, and answer correctness (Gao et al. 2023). A generated citation that was not present in the evidence bundle is invalid. Reauthorize evidence when rendering a saved answer because a once-valid source may have been revoked.

Retrieved documents are untrusted

Treat text, metadata, OCR output, image text, and annotations as untrusted data. An indirect prompt injection in a retrieved page is content, not system policy (Greshake et al. 2023). It cannot grant authority, become a tool instruction, change the tenant, disable citations, or request data exfiltration. Keep instructions and evidence in distinct channels, allow-list tool actions, validate arguments, and apply authorization again at each side effect.

Also test a poisoned corpus: an attacker may insert apparently relevant chunks to steer answers or citations (Zou et al. 2025). Restrict ingestion authority, record source trust and provenance, review unusual source or ranking changes, and retain a known-clean rollback generation. Content scanners help, but the core protection is that retrieved text has no authority of its own.

Evaluate layers and failure modes

One end-to-end score cannot identify where evidence was lost. Keep an oracle for each layer and compare the deployed stage with that oracle.

Use document-family splits rather than randomly placing sibling chunks in train and test. Include exact identifiers, paraphrases, negation, multilingual questions, tables, temporal questions, ambiguous entities, multi-hop evidence, hard negatives, and no-answer cases. Calibrate model judges against a stable human-reviewed sample; frameworks such as RAGAs can help automate measurement, but do not define the product threshold (Es et al. 2024).

The acceptance suite needs negative cases: wrong tenant, deleted document, stale revision, poisoned chunk, malformed table, empty retrieval, duplicate chunk, selective-filter underfill, reranker timeout, index failure, and no-answer behavior. Exercise fallback and rollback for each relevant failure. An outage may reduce ranking quality under a named policy; it may never weaken authorization or fabricate evidence.

Operate the release lifecycle

1. Freeze the contract. Record corpus, query classes, policy, service levels, metrics, thresholds, latency and cost budgets, and rollback criteria.
2. Build a labeled corpus. Stratify documents, queries, relevance judgments, citations, security cases, and expected abstentions.
3. Establish a lexical baseline. It exposes corpus and judgment failures before embedding and ANN complexity arrive.
4. Evaluate candidates. Compare parsing, segmentation, sparse, dense, visual, fusion, reranking, and context policies as controlled ablations.
5. Build and reconcile. Publish only a complete immutable generation with a manifest, provenance, policy, and tombstone coverage.
6. Shadow. Replay production-shaped queries without serving the result.
7. Canary. Route a small authorized cohort and monitor quality, leakage, freshness, latency, cost, and fallbacks.
8. Promote or roll back. Switch atomically; keep and exercise the last-known-good generation.
9. Monitor drift. Sample queries and documents by stratum, investigate changes, and preserve decision traces without logging sensitive payloads by default.
10. Requalify. A parser, segmenter, embedding space, analyzer, index, reranker, policy schema, corpus boundary, or material workload change is a requalification trigger.

The output is a retrieval release record: the immutable manifest, evaluation results, security and failure evidence, freshness and deletion measurements, canary decision, owner, approval, rollback target, and requalification triggers. That record is the handoff to the generator or agent team and the input to the wider evaluation practice in Chapter 87.

Further reading