Model Judges and Preference Rankings

An model-as-judge, a model used as the grader, can turn a written rubric into thousands of repeatable judgments. It cannot turn an underspecified idea of quality into truth. Its verdict depends on the evidence it receives, the rubric and prompt, the judge model and sampling settings, the candidate order, and the population of cases. Change any of them and the measurement may change.

The right question is therefore not “Which judge is best?” It is “For which decision, criterion, cases, and failure costs has this judge been validated?” Human judgments from Chapter 49 remain observations under their own protocol. A model judge is a second instrument calibrated against those observations or against checkable outcomes, not a replacement source of ground truth.

Use the Narrowest Grader That Fits

Open-ended output is not uniformly uncheckable. A response may contain claims that can be verified against supplied evidence, code that can be executed, citations that can be resolved, and style that requires judgment. Decompose the evaluation before choosing a grader:

A model judge is most defensible when the criterion is observable, the necessary evidence fits in its input, and errors can be checked on a representative sample. It is a weak choice when the task requires unavailable facts, tacit domain expertise, or authority the judge does not possess. G-Eval demonstrated that structured criteria and form-filling can improve correspondence with human ratings for summarization and dialogue, while also finding bias toward model-generated text (Liu et al. 2023). That is evidence for a protocol on tested tasks, not a general certificate.

HELM introduced “holistic” evaluation as broad, standardized coverage across scenarios and metrics, with raw prompts and completions retained for analysis (Liang et al. 2023). Holistic does not mean that one judge should compress every property into one score. It means the evaluation preserves the matrix of use cases, capabilities, risks, efficiency, and uncertainty long enough for a real decision rule to use it.

Freeze the Judge as a Versioned Instrument

The judge is more than a model identifier. It is the complete transformation from a case to a verdict:

JudgeSpec:
  judge_model_and_revision
  rubric_revision
  prompt_template_hash
  evidence_policy
  output_schema
  candidate_order_policy
  sampling_parameters
  parser_and_repair_revision
  abstention_and_escalation_rule

JudgeRecord:
  judge_spec_hash
  evaluation_spec_hash
  case_and_candidate_ids
  candidate_order
  evidence_refs
  raw_model_output
  parsed_dimension_labels
  tie_or_abstention_state
  rationale_refs
  latency_cost_and_attempts
  created_at

Pin every field that can change a verdict. Rewording a criterion, upgrading the judge model, changing temperature, or repairing malformed output with a new parser creates a new judge revision. Regrading writes a new record beside the old one; it does not overwrite history.

Keep dimensions separate when their errors and consequences differ. Correctness, citation support, tone, and completeness need distinct criteria and labels even if one API call returns them together. If a release rule later combines them, record the weights, vetoes, and missing-data behavior in the evaluation specification rather than hiding them inside the judge prompt.

The output needs honest uncertainty states. “Tie,” “both fail,” “insufficient evidence,” “criterion not applicable,” and “judge failure” are not interchangeable. A forced winner turns missing information and parser errors into preference data. Preserve the raw response even when parsing fails, and route cases covered by the escalation rule to a qualified person or deterministic checker.

Validate Before Scaling

Develop the rubric, prompt, examples, and parser on development cases. Then freeze the JudgeSpec and evaluate it on a locked set that did not influence those choices. The reference protocol must match the decision: qualified users for user preference, specialists for domain correctness, policy raters for compliance, or executable outcomes where available.

Validation is criterion- and slice-specific. Report at least:

• the human protocol, case population, and independent unit;
• raw agreement and confusion patterns, including ties and abstentions;
• false-pass and false-fail rates where a reference label exists;
• results by criterion, language, domain, difficulty, candidate family, and output style named in advance;
• uncertainty clustered at the actual independent unit;
• sensitivity to judge sampling, evidence removal, and prompt or parser failures.

Zheng et al. reported over 80% agreement between GPT-4 judgments and human preferences in their MT-Bench and Chatbot Arena experiments, alongside position, verbosity, self-enhancement, and reasoning limitations (Zheng et al. 2023). The result establishes performance for those tested protocols. It does not imply 80% accuracy on a new rubric, domain, judge revision, or subject-model distribution.

Agreement is not validity, as the previous chapter showed. A judge can consistently reproduce a flawed adjudication policy. Compare it with independent raw human judgments as well as adjudicated labels, inspect disagreements, and keep executable counterexamples where possible. If the judge emits a probability intended as confidence, assess calibration; a verbal rationale or a numeric score is not automatically a probability.

Measure Residual Bias, Not Prompted Impartiality

“Be unbiased” is not a control. Use transformations that should leave the target judgment unchanged, then measure whether the verdict moves.

Position bias can reverse comparisons even when only response order changes; Wang et al. demonstrated the effect and a calibration framework on their tested judge and response pairs (Wang et al. 2024). Swapping order reveals the problem but does not guarantee its removal. A protocol may require consistent verdicts across both orders, average probabilities when the output has a justified probabilistic interpretation, or retain an explicit inconsistency state.

Length control is likewise an estimand, not a prompt sentence. Length-Controlled AlpacaEval used regression to estimate preferences at zero length difference and showed that rankings can change after the adjustment (Dubois et al. 2024). The adjusted result answers a counterfactual question about equal length; the unadjusted result answers a question about complete outputs, length included. Report which one serves the decision instead of declaring either universally correct.

Self-preference also needs direct testing. Panickssery et al. found that evaluators could recognize and favor their own generations under controlled conditions, with self-recognition related to the strength of the bias (Panickssery et al. 2024). A different-family judge or an ensemble may diversify this error, but neither proves neutrality.

Finally, candidate text is untrusted input. Raina et al. showed that short adversarial phrases could transfer through surrogate judges and inflate scores, with absolute scoring more vulnerable than comparative assessment in their experiments (Raina et al. 2024). Delimit candidate content, deny it instruction authority, use structured outputs and strict parsers, keep attack probes in the regression suite, and avoid making a model judge the sole gate for a high-impact decision.

Turn Pairwise Votes into a Conditional Ranking

An pairwise comparison asks which of two candidates better satisfies a criterion on the same case, with tie and unscorable options. It reduces scale-anchoring burden, but it does not remove rater bias or create an absolute quality level.

Bradley and Terry introduced their paired-comparison model in 1952 for incomplete block designs; modern model leaderboards reuse the same latent-score idea (Bradley and Terry 1952). Arena-style systems aggregate many comparisons with a Bradley-Terry model or a related rating system. For comparison $k$, a useful extension is

$$\Pr(Y_k=1)=\sigma\!\left(\beta_i-\beta_j+\boldsymbol{\gamma}^{\mathsf T}\mathbf{x}_k\right).$$

Here $Y_k=1$ means candidate system $i$ beats system $j$ in comparison $k$; $\beta_i$ and $\beta_j$ are their latent ratings; $\mathbf{x}_k$ is a recorded covariate vector such as order or length difference; $\boldsymbol{\gamma}$ contains the corresponding coefficients; ${\mathsf T}$ denotes transpose, so $\boldsymbol{\gamma}^{\mathsf T}\mathbf{x}_k$ is their inner product; and $\sigma(a)=1/(1+e^{-a})$ is the logistic function. One constraint such as $\sum_i\beta_i=0$ is required because adding the same constant to every rating changes no probability (Bradley and Terry 1952).

The familiar Elo display uses a base-10 parameterization of the same logistic shape. In the common 400-point scale, equal ratings predict a 50% win probability and a 400-point difference predicts about 91%. The scale changes the displayed rating gap, not the underlying evidence.

This model is a compact hypothesis: systems have scalar strengths on the sampled comparison population, and those strengths explain transitive preferences. Before publishing a global rank, check what the hypothesis hides:

• Is the directed win graph strongly connected, so every partition has wins flowing both ways through the graph? Without that condition, an undefeated or separated group can make the unregularized maximum-likelihood ratings diverge (Ford 1957).
• Were prompts, users, languages, and opponent assignments sampled for the target population?
• Are preferences heterogeneous or cyclic in ways one scalar cannot represent?
• Were ties modeled, reported separately, or silently converted into half-wins? Binary Bradley-Terry does not model tie probability; tie-aware extensions add another outcome and parameter (Davidson 1970).
• Did repeated votes from one user, prompt, or conversation get treated as independent?
• Did model availability, selective submission, or retraction change the comparison pool?

Chatbot Arena demonstrated large-scale pairwise evaluation with crowdsourced prompts and votes, and analyzed agreement between crowd and expert raters for its collected population (Chiang et al. 2024). Its rating is therefore a conditional estimate of preference among participating users, prompts, models, and interface conditions, not an intrinsic capability constant.

Fit uncertainty with the assignment intact. Resample the independent prompt or user cluster rather than individual votes when observations repeat within it. Report rating intervals, rank probabilities or sets, vote counts, tie rates, and sensitivity to weighting and eligibility rules. Human-vote and model-judge rankings should remain distinguishable even when they share the same fitting code.

Holistic Evaluation Preserves the Vector

A single preference score cannot answer every deployment question. Keep the result as a scenario-by-metric matrix:

• task success and criterion-level quality;
• factuality and evidence support;
• safety, fairness, and policy guardrails;
• latency, cost, and resource use;
• judge coverage, abstention, and error rates;
• uncertainty and known validity limits.

The release decision may apply a primary objective, non-inferiority margin, and hard guardrails. Another decision may use a Pareto frontier rather than a weighted total. Either is defensible when declared before looking at the candidate results. An undocumented average is not holistic; it merely conceals which failures were traded away.

Protect Confirmation as an Information Boundary

A private test set, an access-controlled confirmation set, reduces direct exposure and repeated tuning. It does not prove that no semantically equivalent case appeared in pretraining, nor does a contamination detector prove absence. Shi et al. formalized black-box pretraining-data detection and demonstrated one method on their benchmark, while noting the underlying problem created by undisclosed training corpora (Shi et al. 2024). Treat such audits as evidence with error, not certificates.

Public, live, and private evaluations serve different purposes. Public suites support reproducibility and shared diagnosis. Live preference systems refresh prompts but inherit the population, interface, submission, and sampling policies of the platform. Private confirmation sets support internal decisions but offer weak external comparability and can still be overfit through repeated queries; adaptive reuse is a statistical problem even when the data never leaves the evaluation service (Dwork et al. 2015).

Maintain the boundary operationally:

1. Separate development cases from confirmation cases and restrict the latter's raw content.
2. Log every evaluation, candidate revision, judge revision, and selection decision.
3. Limit repeated queries and include the full selection history in the final inference.
4. Prevent confirmation outputs and human corrections from silently entering training data.
5. Add fresh cases from production failures to development first; promote a separately collected set for later confirmation.
6. Rotate or retire exposed cases, while retaining old versions for historical audit.

Freshness is not only about secrecy. A newly collected post-cutoff case can strengthen evidence against pretraining exposure; a private copy of an old public benchmark cannot. Conversely, a secret set sampled from the wrong users or tasks remains invalid no matter how well it is protected.

An Operating Contract for Judge-Based Evaluation

Before a judge-based score can gate a release, its record should answer:

1. What decision, construct, case population, and independent unit does it serve?
2. Which properties use deterministic checks, humans, model judges, or observed outcomes?
3. What exact JudgeSpec and evidence policy produced each raw verdict?
4. Which locked reference set validated that revision, and which cases tuned it?
5. What are its criterion- and slice-level false-pass, false-fail, tie, abstention, and parser-failure rates?
6. How sensitive is it to order, length, style, model family, nondeterminism, missing evidence, and adversarial content?
7. How are ties, repeated observations, clustering, weights, and rating uncertainty handled?
8. What confirmation-set access, query, reuse, rotation, and escalation rules preserve independence?
9. Which model, judge, rubric, or domain changes require revalidation?

The handoff to training must remain explicit. Preference labels may train a separate reward model or select examples, so judge errors can shape later behavior. A reward model is not literally the same object as a prompted model judge: it is another learned scoring instrument with its own data, objective, and validation contract. Chapter 19 develops that distinction.

Further reading