Factuality, Grounding, and Evidence

An answer can be true without being grounded, grounded in a mistaken source without being true, and decorated with citations that support none of its claims. These are different failures. Calling all of them “hallucination” hides the decision an evaluation is supposed to inform.

The useful unit is a claim together with its evidence boundary, status, and provenance. The protocol must say which world or source counts, at what time, whether search is allowed, which claims require support, and what happens when the available evidence is insufficient. Only then can a score distinguish model error from retrieval, context, citation, or evaluator error.

Define the Claim Before the Metric

Several related terms answer different questions:

TruthfulQA operationalized truthfulness with 817 questions designed around false beliefs and misconceptions. Its finding that larger models were generally less truthful applies to the models and adversarial questions tested, not to model scaling in general (Lin et al. 2022). Factuality is broader and must name an authority and time: an exchange rate, office holder, or clinical recommendation can change. Faithfulness is narrower. A faithful summary of an erroneous document can be untrue in the world; a true fact supplied from memory can be ungrounded when the task requires use of the given document.

Attribution is not a proof of truth. The Attributable to Identified Sources framework asks whether a statement about the external world is supported by an independent, identified source under explicit annotation guidelines (Rashkin et al. 2023). Source quality and currentness remain separate judgments. Citation quality adds another layer: the interface must connect each check-worthy claim to the span that supports it, not merely list a document at the end.

Use these terms as record fields rather than synonyms. “Unsupported” means the evaluator did not find sufficient support inside its declared evidence boundary. It does not, by itself, mean false. “Contradicted” requires positive conflicting evidence. “Insufficient evidence” must remain distinct from both.

Freeze the Evidence Contract

The evaluator is a versioned pipeline, not a prompt that asks whether an answer is factual. Freeze the full contract before comparing systems:

FactualitySpec:
  decision
  failure_costs
  case_population
  authority
  as_of_time
  mode:
    closed_book
    supplied_context
    bounded_search
  corpus_snapshot
  access_and_source_policy
  claim_policy_and_revision
  retrieval_and_span_policy
  verdict_policy_and_judge
  citation_mapping_policy
  abstention_policy
  aggregation_policy

ClaimRecord:
  case_id
  response_id
  claim_id
  claim_text
  resolved_context
  checkworthiness
  weight
  candidate_evidence_spans
  cited_evidence_spans
  verdict
  extractor_revision
  retriever_revision
  judge_revision
  raw_verdict
  rationale
  adjudication

In closed-book mode, the reference set grades parametric answering; no retriever is part of the tested system. In supplied-context mode, the answer must stay within a fixed context. In bounded-search mode, the corpus, search provider, query date, access rights, and retrieval budget become part of the case. Results from these modes are not interchangeable.

Keep exact document and span identifiers, content hashes, retrieval scores, context order, truncation, candidate text, citation offsets, and raw judge output. Regrading appends a new verdict. It must not erase the evidence or evaluator revision that produced the old one.

Decompose Without Losing Coverage

Evidence-based fact verification predates current language models. FEVER labeled claims against a fixed Wikipedia snapshot as supported, refuted, or not enough information; supported and refuted cases include evidence sentences, while the last label does not assert falsity (Thorne et al. 2018). FActScore adapted the claim-level idea to long-form generation in 2023 by decomposing biographies into atomic facts and measuring factual precision against a knowledge source (Min et al. 2023).

Here $y$ denotes an answer, and the extraction policy produces the claim sequence $C(y)=(c_1,\ldots,c_m)$. The unweighted FActScore form is

$$F_K(y)= \frac{1}{m}\sum_{i=1}^{m}\mathbf{1}[S(c_i,K)=1].$$

Here $F_K(y)$ is the FActScore of answer $y$ relative to knowledge source $K$; $C(y)$ is the answer's sequence of extracted atomic claims; $c_i$ is claim $i$; $m$ is the number of extracted claims; $S(c_i,K)=1$ means the protocol finds $c_i$ supported by $K$, and $S(c_i,K)=0$ otherwise; and $\mathbf{1}[\cdot]$ is the indicator function. An automated implementation must record evidence retrieval separately from the support label. When $m=0$, the score is undefined, not a perfect result. Eligibility or task-completion logic must handle an answer with no checkable content.

An atomic claim is a self-contained statement conveying one independently checkable piece of information, after resolving pronouns, dates, units, and local context. Atomicity is a protocol choice, not a unique linguistic truth. “The trial enrolled 400 patients and reduced mortality” contains at least two claims; “it happened the following year” cannot be checked after its antecedent and date are discarded.

Claim extraction is itself a measurement step. Audit whether the extractor omits hedged claims, turns one conjunction into several easy wins, duplicates facts, or changes boundaries across writing styles and languages. Preserve the original span and parent relation so an adjudicator can reconstruct the answer. Measure extraction recall on human-marked samples and test whether reasonable re-splitting materially changes the result.

Factual precision alone rewards saying less. Report the supported fraction beside the number and weight of check-worthy claims, requested-content coverage, contradiction rate, and response eligibility. SAFE made this trade-off explicit by combining supported-fact precision with a user-chosen target for the number of facts. On the LongFact experiment, SAFE agreed with crowd labels on 72% of 16,011 already human-decomposed facts. In a separate audit of 100 disagreements, researchers with broader web access preferred its verdict 76% of the time; the reported evaluator was also more than 20 times cheaper under its 2024 model and search prices (Wei et al. 2024). These figures do not validate end-to-end claim extraction or establish that search agents are generally more accurate than people.

Validate the Evaluator Before the Model

A claim scorer can fail at extraction, evidence search, source selection, or the final support judgment. Validate each stage on locked, independently adjudicated cases. Report claim-extraction misses, evidence-retrieval misses where exhaustive gold evidence exists, and the verdict confusion matrix for supported, contradicted, insufficient, and not-checkable claims. Break results down by domain, language, temporal sensitivity, numerical reasoning, multi-source claims, negation, and candidate style.

The verifier must use the declared evidence, not silently supplement it with parametric memory. Include adversarial cases in which the candidate instructs the evaluator to ignore its rubric, a citation points to a topically similar but non-supporting passage, evidence conflicts, or the correct outcome is insufficient evidence. As in Chapter 50, pin the judge and parser, retain abstentions and failures, compare model verdicts with raw human judgments, and revalidate after any material change.

Do not infer root cause from a final label alone. An unsupported claim may reflect a corpus gap, retrieval miss, packing loss, generator invention, bad citation attachment, extractor error, or verifier false negative. Root-cause attribution requires the recorded trace.

Treat Abstention as a Decision Policy

Short answers remove the decomposition problem but not the decision problem. SimpleQA was designed around 4,326 short fact-seeking questions intended to have one indisputable answer, with grades of correct, incorrect, or not attempted (Wei et al. 2024). “Not attempted” describes the observed response; it does not prove the model recognized its own uncertainty. The automatic grader was checked manually on 300 sampled completions, but the paper did not report a formal grader study. The benchmark therefore tests one adversarially collected, closed-book question population and one grading protocol, not factuality in every form. A later SimpleQA Verified preprint created a filtered 1,000-question derivative after auditing labels, duplicates, and topic balance; it did not repair the original dataset in place (Haas et al. 2025).

Rejection has a longer history than language-model evaluation. Chow formalized the error-reject trade-off for pattern recognition in 1970 (Chow 1970). When a system emits a candidate answer with confidence $p_n$, and the deployment releases it only above threshold $t$, report both coverage and selective risk:

$$\begin{aligned} \operatorname{Coverage}(t) &= \frac{1}{N}\sum_n a_n(t), \\ \operatorname{Risk}(t) &= \frac{\sum_n a_n(t)(1-z_n)}{\sum_n a_n(t)}. \end{aligned}$$

Here $N$ is the number of evaluation cases; $n$ indexes a case and each sum runs from $n=1$ through $N$; $p_n\in[0,1]$ is the reported confidence in its candidate answer; $t\in[0,1]$ is the locked release threshold; $a_n(t)=\mathbf{1}[p_n\ge t]$ records release; and $z_n=1$ when the candidate answer is correct or $0$ when it is wrong. Coverage is the released fraction. Selective risk is the error rate among released answers and is undefined when the system releases nothing. Overall correct rate, $N^{-1}\sum_n a_n(t)z_n$, must remain visible so refusing every case cannot look successful. A system that emits refusals directly needs those refusals labeled rather than assigned invented confidence values.

Plot risk across the coverage range rather than choosing a threshold after seeing the test result. The deployed threshold belongs to a cost policy: a medical assistant, a trivia tool, and an internal search box do not have the same cost for error, refusal, or escalation. Calibration and selection are related but distinct. If random variables $P$ and $Z$ denote reported confidence and candidate-answer correctness on the target population, probability calibration requires $\Pr(Z=1\mid P=p)=p$ for relevant values $p$. Check that relation before applying the abstention threshold; calibration only on released answers describes a selected subset and can hide failures below the threshold.

Diagnose the Grounding Pipeline

Retrieval-augmented generation adds failure surfaces; it does not make an answer grounded by construction. Test each stage with the artifact it actually receives:

If gold evidence is not exhaustive, do not call a model-judged relevance score retrieval recall. Report it as a relevance estimate under that judge. Likewise, finding a supporting passage after generation shows that support is available; it does not prove the generator causally used that passage.

RAGAS proposed reference-free proxy metrics for faithfulness, answer relevance, and context relevance, using model and embedding judgments rather than human reference answers (Es et al. 2024). The archival paper did not define context recall. ALCE evaluates retrieval-backed answers along fluency, task-specific correctness, citation recall, and citation precision; its automatic citation judgments are themselves NLI-model estimates (Gao et al. 2023). The original 2025 FACTS Grounding benchmark used long documents of up to 32,000 tokens, a separate eligibility gate, and an aggregate of three model judges to score response-level grounding (Jacovi et al. 2025). Treat that design as a versioned benchmark contract. A panel can reduce dependence on one judge, but it does not turn the verdict into ground truth.

Score Citations as Claim-Source Links

Citation completeness and citation correctness answer different questions. Here $I$ is the set of claims that require external support; $c_i$ is claim $i\in I$; $w_i>0$ is its declared importance weight; $A_i$ is the set of source spans attached to $c_i$; and $Q(A_i,c_i)=1$ when those spans jointly provide sufficient support without contradiction, or $0$ otherwise. Then

$$C_{\mathrm{cite}}= \frac{\sum_i w_i Q(A_i,c_i)}{\sum_i w_i}.$$

For link-level correctness, let $L$ be the set of attached claim-span links $(i,j)$, where $j$ identifies one cited span, and let $q_{ij}=1$ when span $j$ contributes support to claim $c_i$ without contradicting it, or $0$ otherwise:

$$P_{\mathrm{link}}= \frac{\sum_{(i,j)\in L}q_{ij}}{|L|}.$$

Here $C_{\mathrm{cite}}$ denotes citation completeness, and each $\sum_i$ runs over $i\in I$. The value $P_{\mathrm{link}}$ denotes link correctness, and $|L|$ is the number of citation links. If $I$ or $L$ is empty, the corresponding metric is not applicable unless the protocol independently establishes that no citations were required. Report contradiction, incorrect placement, stale sources, and source quality separately. A response can have high link correctness and low completeness by citing one claim perfectly while leaving the rest unsupported.

Use Weak Signals Only for Triage

Signals that do not consult authoritative evidence can prioritize review, but they cannot certify truth. SelfCheckGPT sampled a black-box model repeatedly and used sentence-level inconsistency as a warning signal on a WikiBio experiment (Manakul et al. 2023). Consistent samples may repeat the same common falsehood; inconsistent samples may reflect ambiguity or several valid answers. Citation count, URL validity, lexical overlap, judge confidence, and agreement among related models have the same limitation: each can reveal a suspicious case, but none establishes factual support.

An Operating Contract for Evidence Evaluation

Before a factuality or grounding result can gate a release, its record should answer:

1. Which decision, population, authority, as-of time, and error costs define the target?
2. Is the system closed-book, supplied-context, or bounded-search, and what exact evidence boundary applies?
3. Which claims require support, how are claims extracted, and how is extraction recall audited?
4. Which statuses distinguish support, contradiction, insufficient evidence, and non-checkable content?
5. Which corpus, retriever, ranking, packed-context, generator, and citation artifacts are retained?
6. How were extractor, retriever, verifier, and model judge validated on locked cases and slices?
7. Which metrics expose factual precision, requested-content coverage, contradiction, abstention, and citation quality without hiding omissions?
8. How are temporal change, source conflict, access control, and source quality handled?
9. Which failures trigger human review, deterministic checking, retrieval repair, or refusal?
10. Which data, model, prompt, corpus, policy, or judge changes require revalidation?

The handoff to Chapter 46 and Chapter 44 is the complete trace, not just a score. The handoff to Chapter 53 is an append-only regression case containing the evidence boundary and every stage needed to reproduce the failure.

Further reading