Human Evaluation and Rubrics

The human label is not a primitive. It is the output of a protocol: who was asked, what they were shown, what rubric they followed, how disagreement was handled, and which cases were later audited. If that protocol is weak, calling the result "human judgment" only hides the weakness.

This chapter sits between statistics and model judges. It treats human evaluation as an instrument that must be designed, calibrated, and versioned before it can train rewards in Chapter 19, validate judges in Chapter 50, or gate releases in Chapter 53.

The Protocol Creates the Label

The phrase "human preference" sounds as if the label were already in the annotator's head. It is not. The label is created by a chain of choices: sampling, task framing, interface, rubric, worker selection, training, quality control, aggregation, and adjudication. A model answer does not receive a human label in the abstract. It receives a label under one of these protocols.

This is why human evaluation does not automatically solve evaluation. Karpinska et al. found that crowd workers, even with filters, can fail to distinguish machine-written and human-written open-ended stories, while English teachers and paired examples provide a stronger signal (Karpinska et al. 2021). Clark et al. reached a similar warning from another direction: as generated text becomes fluent, untrained evaluators may distinguish GPT-3 text from human text only at chance level (Clark et al. 2021). A human may be the right measuring instrument, but the instrument still needs a measurement design.

For LLM systems, the design begins with the decision being served. A broad model comparison can run on ordinary users giving pairwise preference. When the task is medical summarization, the release needs domain experts and a set of factual error categories. A safety gate raises the bar again, calling for adversarial raters and a harm taxonomy. For a product task the strongest raters are not people staring at text at all, but users performing the downstream workflow. The rater population is part of the claim. HealthBench shows this design at scale: 262 physicians with practice experience across 60 countries wrote per-conversation rubrics, 48,562 criteria in all, and the model judge that applies them is itself graded against physician grading before its scores count (Arora et al. 2025).

Rubrics Are Interfaces for Judgment

A rubric is not decoration around a score. It is the measurement contract. It tells raters what counts as a defect, what to ignore, what to do when dimensions conflict, and how to handle uncertainty. Weak rubrics collapse many properties into one vague word, usually "quality." Strong rubrics separate dimensions that should not trade silently against each other.

A useful LLM rubric usually has at least four design choices:

• Decision axis. The property being judged: correctness, helpfulness, faithfulness, safety, tone, concision, or task success.
• Unit. The object being judged: answer, claim, citation, turn, full dialogue, tool call, or final state.
• Scale. Binary pass/fail, ordered levels, pairwise preference, or continuous score.
• Evidence rule. What the rater is allowed to use: the model output only, retrieved context, ground-truth answer, execution trace, or production state.

These choices should be versioned. A rubric that changes from "prefer complete answers" to "prefer concise answers" can reverse a ranking without any model changing. In Chapter 50 the same issue reappears for model judges: a judge prompt is a rubric encoded as an evaluation program.

Rubrics also decide whether the evaluation can later become training data. In Chapter 19, demonstrations and pairwise preferences are used to train reward models, models that learn to score an output the way humans would so the score can stand in for a person during training. If the original rubric rewarded verbosity, the reward model inherits that preference. If it failed to separate factuality from politeness, the model may learn to be agreeable at the expense of being correct. Human evaluation is upstream of behavior, not merely downstream of it.

Agreement Is a Property to Measure

Raw agreement is a poor reliability measure because easy labels can inflate it. If 95% of examples are good and every rater says "good," raw agreement is 95%, but the labels contain little information. Cohen's $\kappa$ adjusts two-rater agreement for chance agreement (Cohen 1960):

$$\kappa = \frac{p_o - p_e}{1 - p_e},$$

where $p_o$ is observed agreement and $p_e$ is expected agreement under the raters' marginal label rates, that is, each rater's overall rate of saying "good" regardless of the example. A $\kappa$ near zero says the raters are not adding much beyond their base rates, even if raw agreement looks high.

Many LLM evaluation tasks use more than two raters, missing labels, ordinal scales (ranked levels with no fixed spacing between them), or uneven assignment. Krippendorff's $\alpha$ generalizes the same idea by comparing observed disagreement with disagreement expected by chance (Hayes and Krippendorff 2007):

$$\alpha = 1 - \frac{D_o}{D_e}.$$

Here $D_o$ is observed disagreement and $D_e$ is expected disagreement. The exact distance function can be nominal, ordinal, interval, or ratio, which makes $\alpha$ a better fit for heterogeneous rubric data than a single two-rater coefficient.

Agreement numbers do not prove a rubric is correct. They prove that trained raters can apply it consistently. That is still valuable. A low agreement score means the evaluation cannot safely drive a release gate or a reward model until the rubric is clarified, the interface improved, or the task split into cleaner dimensions.

Pairwise, Absolute, and Task-Based Evaluation

The common formats have different failure modes.

Pairwise comparison asks which of two outputs is better. It is easier for humans than absolute scoring and feeds naturally into Bradley-Terry or Elo ranking, methods that convert a pile of win/loss votes into a single rating per model, as Chapter 50 does for model judges. Its weakness is that it yields preference, not calibrated quality. A model can win pairwise because it is longer, smoother, or more confident, the same biases that Chapter 50 has to correct for model judges.

Absolute scoring asks for a grade on a fixed scale. It is cheaper to aggregate and easier to attach to a threshold, but raters interpret scale points differently. A "4" from one expert may be a "3" from another. Anchored examples and calibration rounds are mandatory, not optional.

Task-based evaluation gives the generated artifact to a person doing the real downstream work. It is often the strongest evidence because it measures utility rather than surface preference. It is also expensive, slow, and domain-specific. That cost is why most teams fall back to preference or rubric scores even when the product claim is about task completion.

GENIE showed how much the interface and quality-control layer matter for reproducible text-generation evaluation (Khashabi et al. 2021). The deeper lesson is that no format is intrinsically gold. The format is gold only relative to the claim it supports.

From Human Labels to Automated Judges

Human evaluation becomes infrastructure when it calibrates other instruments. A small expert-labeled set can validate a model judge. A periodic audit can catch judge drift. A disagreement set can reveal which rubric dimensions remain underspecified. Fernandes et al. describe this broader feedback loop: human feedback is not a single object, but a family of signals used for training, decoding, evaluation, and AI-feedback systems (Fernandes et al. 2023).

The handoff to a model judge should be explicit:

1. Humans label a representative and adversarial sample under a versioned rubric.
2. Reliability is measured before labels are treated as ground truth.
3. A model judge is calibrated against that labeled sample, including abstention cases.
4. The judge is used for scale, but only inside the task distribution where calibration held.
5. Fresh human audits periodically remeasure agreement between judge and humans.

This is the same discipline as Chapter 48, applied to a human instrument. A judge that agrees with humans on a narrow writing task may fail on legal analysis. A crowd label that works for casual chat may fail for biology. The unit of trust is not "humans" or "GPT-4 as judge." The unit of trust is a protocol, a population, a rubric, and a measured agreement on a named distribution.

Further reading