Retrieval and Document Intelligence

Retrieval is how a model talks to data it was never trained on, and document intelligence is the front door that decides what data even reaches it. This practice layer sits under the theory in Chapter 44 and Chapter 46: those chapters explain why retrieval works and how a context window is assembled; here the work is choosing 2026 tools and wiring them together. The two halves are one pipeline. A PDF becomes structured text, structured text becomes embeddings in a vector store, a query pulls the right chunks back, and a reranker orders them before they hit the prompt. A failure anywhere upstream silently poisons everything downstream.

The pipeline and why parse quality leads

Read the pipeline in ingestion order. Figure 86.2 shows the whole path from a raw document to an answer.

The order matters because the largest data loss and the quietest errors happen at the parse step, not the search step. If a two-column page is flattened into interleaved lines, or a financial table is shredded into prose, no embedding model and no reranker downstream can recover the structure. By mid-2026 the vector database itself is close to a commodity: approximate nearest-neighbor search (HNSW, IVF, DiskANN variants) is everywhere and rarely the bottleneck. The differentiation has moved to the parse quality at the front, the embedding and reranker quality in the middle, and the cost architecture (RAM-resident versus object-storage) underneath. We treat the two halves in pipeline order: document intelligence first, then the retrieval stack.

Document intelligence: getting from a messy file to clean data

Document intelligence turns a PDF, scan, screenshot, or office file into machine-readable structure: text in reading order, tables as cells, equations as LaTeX, key-value pairs, and ultimately a typed object that obeys a schema. By 2026 the field split into three architectural camps, plus a cross-cutting concern.

1. Classic CV and optical character recognition (OCR) pipelines. Here OCR means recognizing text from pixels, inside a multi-stage stack: layout detection, OCR, table reconstruction, reading-order assembly. Deterministic, debuggable, fast, and brittle on unusual layouts. Marker, MinerU, PaddleOCR, unstructured, and the cloud APIs live here.
2. End-to-end document VLMs. A single image-and-text-to-text model emits the whole structured document (Markdown, HTML tables, LaTeX) in one pass. These dominate the open leaderboards, are usually small (sub-2B), and are self-hostable on the same serving fleet as your chat models (Chapter 82). Simpler to run, harder to debug when they drop a column.
3. Agentic document platforms. A managed service wraps OCR, a vision-language model (VLM), a model that reads page images and text together, a correction loop, and structured extraction behind an API and an SLA. You trade self-hosting for accuracy on hard documents and for compliance.

The cross-cutting concern is structured extraction: going from parsed text to a typed object that obeys a JSON Schema. That is where constrained decoding and validation libraries meet the parse.

The table below is a condensed view; the Primary sources hold the version and license detail.

One option is missing from the table on purpose. A general frontier VLM such as Gemini Flash or Pro parses pages well with no pipeline at all, and it is the baseline in most 2026 comparisons. It costs more per page than a dedicated small OCR VLM and can hallucinate or silently drop content on dense tables, so treat "just send the page to a frontier VLM" as a real, simple option that you benchmark against the specialists, not as a default.

A more radical design skips the parse entirely. ColPali-style visual retrieval embeds page images with a vision-language retriever, scores them with the same late-interaction matching the retrieval section below describes for text, and hands the retrieved pages to a VLM generator that reads the pixels (Faysse et al. 2024); Cohere Embed v4's image mode serves the same pattern with single-vector embeddings. By 2026 this is a standard alternative for visually rich corpora, and it qualifies the claim that parse quality leads: skip the parse and nothing upstream caps the retriever. The costs move rather than vanish. Page-image embeddings cost more to store and query, the generator must be a VLM, and structured extraction, provenance-carrying citation, and any text reuse of the corpus still need a parse.

Structured extraction

Once a page is parsed, turning it into a typed object is an orthogonal choice along one axis: do you control the decoder?

Choosing a parser

The decision is four axes: document difficulty, self-host versus API, volume and cost, and compliance.

• Self-hosted document VLM (PaddleOCR-VL, dots.ocr, DeepSeek-OCR, Granite-Docling, MinerU) when you can run a GPU, want no per-page bill, need data to stay in your VPC, and your documents are "normal hard": academic papers, reports, mixed text and tables. Serve them on vLLM/SGLang and you get constrained decoding for free.
• Docling toolkit as the safest pure-OSS default: a permissive pipeline with RAG-friendly output and the option to drop in Granite-Docling.
• Mistral OCR 3 when you want a hosted API with a flat low per-page price (reported around $2 per 1k pages, roughly half with batch, per Mistral, Dec 2025) and an EU vendor.
• Reducto or LlamaParse Agentic when documents are genuinely adversarial and accuracy matters more than per-page cost, and you want the correction loop, SLA, and compliance without building them.
• unstructured when the problem is format sprawl rather than the single hardest PDF.
• Cloud doc-AI (Azure, AWS, Google) when you are already in that cloud or need a specific prebuilt extractor (invoice, receipt, ID, lending).

A self-host parse plus schema-valid extraction is one request when constrained decoding lives inside the server:

# vLLM / SGLang OpenAI-compatible call: schema-constrained extraction
resp = client.chat.completions.create(
    model="paddleocr-vl",                 # served locally on vLLM
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": page_png_data_uri}},
        {"type": "text", "text": "Extract the invoice as JSON."}]}],
    extra_body={"guided_json": Invoice.model_json_schema()},  # Outlines / XGrammar
)

The retrieval stack: stores, embeddings, rerankers, hybrid search

An embedding (or vector) is a list of numbers that captures a text's meaning, so that texts with similar meaning sit near each other in that number space; the whole stack rests on that one idea. Around the vector store sit three other pieces: embedding models that turn text (and increasingly images and PDFs) into vectors, rerankers that re-score a candidate set for precision, and RAG frameworks that wire ingestion, chunking, retrieval, fusion, and prompt assembly together.

Vector stores

The first question is almost always: do you already run Postgres? The table is condensed; the Primary sources hold the benchmark and pricing detail, all of it vendor-reported and version-sensitive.

The trade-offs underneath those picks are durable even as the products churn. One system versus vector-native: Postgres-with-vectors removes an entire moving part (ops, backups, consistency, joins) at some cost in peak ANN throughput, and for most teams that trade is correct. RAM versus object storage: classic vector DBs keep the working set in RAM (fast, expensive), while object-storage-first designs (Turbopuffer, disk-resident DiskANN in pgvectorscale and Milvus) trade some latency for much lower cost on large or cold corpora. Managed versus self-hosted: managed buys zero ops and predictable scaling, self-hosted buys cost control, data residency, and no per-query markup. Filtering at scale: if most queries are "vector search but only within this tenant or date range or permission set," filtered recall and filtered latency matter more than raw ANN speed, and Qdrant (ACORN) and pgvectorscale (label filtering) are built for it.

Embeddings and rerankers

The database returns what the embedding model encoded, and the reranker fixes the ordering. Spending here usually beats switching databases. Treat the embedding model as a first-class versioned dependency: changing it means re-embedding the whole corpus.

MTEB and MMTEB scores below are not directly comparable across boards or versions (the English board, the multilingual board, and legacy versus current task sets use different task counts), so a higher number is not automatically a better model for your use case. The notes here are directional.

Hybrid search and reranking: the 2026 consensus

The default retrieval strategy in 2026 is hybrid plus rerank.

• Hybrid search runs BM25 (sparse, exact-term) and dense vector search in parallel and fuses the two ranked lists, typically with Reciprocal Rank Fusion: score = sum 1 / (k + rank), with k = 60 by convention. BM25 catches exact matches (codes, names, rare terms) that dense vectors blur; dense catches paraphrase and concept that BM25 misses. Fusing by rank sidesteps the incompatible-score-scale problem.
• Reranking takes the top 50 to 100 fused candidates and re-scores them with a cross-encoder that reads query and document together, returning the top 5 to 10 into the prompt. Late-interaction models (ColBERT-style, MaxSim) sit between dense retrieval and cross-encoder reranking: token-level matching with precomputable document representations, scoring a document by summing each query token's best match against the document's tokens.
• Optionally de-duplicate with MMR (Maximal Marginal Relevance, which picks each next chunk for relevance minus overlap with the chunks already chosen) before reranking so the context window is not spent on near-identical chunks.

A minimal pgvector hybrid retrieval, with the embed and rerank calls going through the gateway (see Wiring):

-- chunks(id, doc_id, content, embedding vector(1024), tsv tsvector)
WITH dense AS (                          -- dense candidates (HNSW / DiskANN)
  SELECT id, RANK() OVER (ORDER BY embedding <=> :query_vec) AS r
  FROM chunks ORDER BY embedding <=> :query_vec LIMIT 100
),
sparse AS (                              -- sparse candidates (BM25-style)
  SELECT id, RANK() OVER (ORDER BY ts_rank_cd(tsv, plainto_tsquery(:q)) DESC) AS r
  FROM chunks WHERE tsv @@ plainto_tsquery(:q) LIMIT 100
)
SELECT c.id, c.content,                  -- reciprocal rank fusion, k = 60
       COALESCE(1.0/(60+dense.r),0) + COALESCE(1.0/(60+sparse.r),0) AS rrf
FROM chunks c
LEFT JOIN dense  ON dense.id  = c.id
LEFT JOIN sparse ON sparse.id = c.id
WHERE dense.id IS NOT NULL OR sparse.id IS NOT NULL
ORDER BY rrf DESC
LIMIT 50;  -- hand these 50 to the reranker, keep the top 5-10 for the prompt

Try changing k below: a small k lets the top of each list dominate, a large k flattens the weighting so agreement across both lists matters more.

# Reciprocal Rank Fusion: score = sum 1 / (k + rank), rank starting at 1.
bm25  = ["d3", "d1", "d7", "d2"]   # sparse, exact-term ranking
dense = ["d2", "d3", "d5", "d1"]   # dense, semantic ranking
k = 60

def rrf(lists, k):
    score = {}
    for ranked in lists:
        for rank, doc in enumerate(ranked, start=1):
            score[doc] = score.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(score.items(), key=lambda kv: kv[1], reverse=True)

for doc, s in rrf([bm25, dense], k):
    in_both = doc in bm25 and doc in dense
    print(f"{doc}  rrf={s:.5f}  {'(in both lists)' if in_both else ''}")
# d3 leads: never #1 in either list, but solidly ranked in both.

Frameworks

For the orchestration layer, the common 2026 pattern is LlamaIndex (MIT) for ingestion and retrieval, LangGraph (MIT) for orchestration and agents when the app is agentic, Haystack (Apache-2.0) when you want explicit testable production pipelines, and DSPy when you want to compile and optimize the retrieval-plus-prompt rather than hand-tune it. Wire in RAGAS or LangSmith from day one. Retrieval without evaluation is guessing, which is exactly the case the eval-practice chapter makes (Chapter 87).

A sensible default (2026)

For a typical team starting this stack in mid-2026 (opinionated, dated 2026-06):

• Document intelligence: a two-tier setup. Default tier is self-hosted OSS: Docling as the pipeline with a small open document VLM (PaddleOCR-VL, or Granite-Docling for the most permissive license) served on vLLM, paired with Outlines/XGrammar guided_json for schema-valid extraction. Escalation tier routes the pages your eval flags as low-confidence (dense tables, handwriting, bad scans) to Reducto or LlamaParse Agentic, or uses Mistral OCR 3 as a cheap hosted fallback. If you cannot run a GPU at all, make Mistral OCR 3 the default and skip the OSS tier.
• Vector store: pgvector on your existing Postgres, adding pgvectorscale when you cross into tens of millions of vectors. If you are not on Postgres, Qdrant is the default vector-native alternative; if you refuse to run any infra, Pinecone or Turbopuffer (the latter when cost at scale or many tenants dominates).
• Embeddings: start with a strong API model (text-embedding-3-large for the safe path, Cohere Embed v4 for multimodal/long context, Voyage 4 on MongoDB Atlas), and go open weight (Qwen3-Embedding or BGE-M3) when you need self-hosting, data residency, or non-English depth.
• Retrieval strategy: hybrid (BM25 + dense) with RRF, then a reranker. Add the reranker (Cohere Rerank 4 managed, or mxbai-rerank v2 self-hosted) before you tune anything exotic. It is usually the highest-ROI single change.
• Framework: LlamaIndex for ingestion and retrieval, LangGraph for orchestration if agentic, and RAGAS or LangSmith wired in from day one.

The rationale is to minimize new infrastructure, put the quality budget where it pays (parse quality, embeddings, hybrid, rerank), and keep a clean upgrade path. You can swap the store later without rewriting the pipeline, but you cannot un-poison answers from a parser and retriever you never measured.

Wiring

Retrieval sits below the agent or generator and beside the model gateway, and document intelligence sits one step further out at the ingestion edge. The unifying seam is the gateway. Embedding models, rerankers, and self-hosted document VLMs are all just served models (Chapter 82), so ideally you call them through the model gateway alongside chat traffic. Then embedding, rerank, and parse calls are keyed, budgeted, and observed exactly like chat calls, and swapping text-embedding-3-large for a self-hosted bge-m3, or PaddleOCR-VL for Mistral OCR on hard pages, is a routing change rather than a code change. The end-to-end wiring of the whole stack is the subject of Chapter 88; here the point is narrower. Parse feeds extraction and chunking; both land in the vector store, which the retrieval step queries; retrieval then hands context to the generator or agent, while evaluation watches both the retrieval and the final answer. A bad parse poisons every layer above it, which is why parse quality is a retrieval and agent concern, not a preprocessing afterthought.

Primary sources

Vector stores:

• pgvector: https://github.com/pgvector/pgvector
• pgvectorscale / StreamingDiskANN: https://github.com/timescale/pgvectorscale; Tiger Data benchmark (vendor-reported) https://www.tigerdata.com/blog/pgvector-is-now-as-fast-as-pinecone-at-75-less-cost
• Qdrant: https://qdrant.tech/documentation/; hybrid search https://qdrant.tech/articles/hybrid-search/
• Weaviate: https://weaviate.io/blog/hybrid-search-explained
• Milvus / Zilliz: https://milvus.io/docs
• Pinecone cost model: https://docs.pinecone.io/guides/manage-cost/understanding-cost; Dedicated Read Nodes https://www.pinecone.io/blog/dedicated-read-nodes/
• Turbopuffer: https://turbopuffer.com/blog/turbopuffer; pricing log https://turbopuffer.com/docs/pricing-log
• LanceDB: https://lancedb.github.io/lancedb/; Chroma: https://docs.trychroma.com/; Vespa: https://docs.vespa.ai/

Embeddings and rerankers:

• MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard (scores differ across boards and versions; not directly comparable)
• Voyage 4 family: https://blog.voyageai.com/2026/01/15/voyage-4/
• Cohere Rerank: https://docs.cohere.com/docs/rerank-overview; Rerank 4 (VentureBeat) https://venturebeat.com/ai/coheres-rerank-4-quadruples-the-context-window-to-cut-agent-errors-and-boost
• Cohere Embed: https://docs.cohere.com/docs/cohere-embed; OpenAI embeddings: https://platform.openai.com/docs/guides/embeddings
• Gemini Embedding 2 (natively multimodal, March 2026): https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-embedding-2/
• BGE-M3: https://huggingface.co/BAAI/bge-m3; Qwen3-Embedding: https://huggingface.co/Qwen; mxbai-rerank v2: https://www.mixedbread.com/

Hybrid search and RAG frameworks:

• Hybrid search / RRF / reranking overview: https://www.digitalapplied.com/blog/hybrid-search-bm25-vector-reranking-reference-2026
• ColBERT / late interaction (survey): https://arxiv.org/pdf/2502.14822; Agentic RAG survey: https://arxiv.org/pdf/2501.09136
• LlamaIndex: https://docs.llamaindex.ai/; LangGraph: https://langchain-ai.github.io/langgraph/; Haystack: https://docs.haystack.deepset.ai/; DSPy: https://dspy.ai/; RAGAS: https://docs.ragas.io/

Document intelligence:

• Docling / Granite-Docling: https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion, https://huggingface.co/ibm-granite/granite-docling-258M/blob/main/README.md
• unstructured: https://docs.unstructured.io/open-source/introduction/overview
• LlamaParse / LlamaCloud pricing: https://www.llamaindex.ai/pricing
• Reducto: https://reducto.ai/pricing, https://reducto.ai/blog/reducto-series-b-funding
• Mistral OCR 3: https://mistral.ai/news/mistral-ocr-3/, https://mistral.ai/pricing/
• OmniDocBench: https://github.com/opendatalab/OmniDocBench; "OmniDocBench is saturated" (Jerry Liu): https://www.llamaindex.ai/blog/omnidocbench-is-saturated-what-s-next-for-ocr-benchmarks
• PaddleOCR-VL / GLM-OCR scores: https://huggingface.co/PaddlePaddle/PaddleOCR-VL/discussions/29, https://arxiv.org/html/2603.10910v1
• Marker / Surya (Datalab) licensing: https://github.com/datalab-to/surya
• Cloud doc-AI pricing: https://aiproductivity.ai/blog/document-ai-cost-comparison/
• Structured extraction (Instructor / BAML / Outlines / XGrammar): https://techsy.io/en/blog/best-llm-structured-output-libraries

Further reading