Human Preference Evaluation

For an explanation, a summary, or a piece of writing, several answers can be correct while one is much more useful to the reader. A fixed answer key captures only part of that difference. Pairwise preference evaluation asks people to compare two responses to the same prompt.

The resulting score describes preferences under the sampled prompts, raters, and instructions. Choosing those samples determines whose needs the comparison represents.

This follows tool-agent evaluation. There, the task often supplied an executable success criterion. Here, the criterion includes human judgment, so we need to specify who rates the responses and how.

What a pairwise comparison records

A useful comparison hides model identities, randomizes response position, and gives the rater a clear task. It should allow ties or another explicit way to record no meaningful preference. A rater who values concise explanations may disagree with one who wants a worked example; neither vote becomes a transcription error just because they differ.

The Chatbot Arena research platform collects anonymous pairwise comparisons on user prompts and uses them to estimate relative model strength. Its participants and prompts are not a random sample of every potential user. The resulting ranking describes that evaluation population, subject to the platform’s filtering and statistical procedure.

For a product-specific evaluation, sample prompts from the use you care about and recruit appropriate raters. Keep categories such as code assistance and factual explanation visible when their requirements differ. A single aggregate can conceal a model that helps one group and frustrates another.

From votes to a rating

The Bradley–Terry model assigns a scalar strength to each competitor and models pairwise preferences from differences in strength. For decisive A/B comparisons, on the conventional Elo scale,

Here $\sigma(x)=1/(1+e^{-x})$. The factor $\ln 10$ is necessary when switching from base-10 Elo notation to a natural-exponential sigmoid. A 100-point difference corresponds to about 64% predicted preference; 400 points corresponds to about 90.9%.

Only differences are identifiable. Adding 500 points to every model changes no pairwise probability. The chosen rating baseline is a convention, not an absolute unit of capability.

Online Elo updates a rating after each comparison:

where $S_A$ is 1 for a win, 0 for a loss, or 0.5 for a tie, and $E_A$ is the current expected score. The other participant receives the opposite update. A larger $K$ reacts more quickly and leaves larger fluctuations.

Arena-style Bradley–Terry fitting estimates strengths from the collected comparison data rather than merely choosing a small Elo $K$ and replaying votes. Tie handling, regularization, sampling weights, and uncertainty estimation are part of the implementation. State those choices when comparing rating systems.

This example computes one online Elo update.

A single scalar ranking is also an approximation. Preferences can vary by task and rater, and can be nontransitive. Inspect subgroup results and model fit rather than assuming every comparison is explained by one underlying number.

Agreement and uncertainty

Agreement is a property of the raters, examples, and rubric being studied. Low agreement can reflect ambiguous instructions, small quality differences, legitimate taste differences, or inconsistent annotation.

Disagreement does not impose a fixed minimum detectable win rate. A small preference difference can be estimated with enough informative observations. Conversely, many repeated votes on similar prompts may add less information than their count suggests.

Arena’s multi-model setting also involves the comparison graph: which systems faced which opponents and on what prompts. Bootstrap procedures or other uncertainty estimates need to respect the data collection process. Overlapping individual rating intervals are not a complete significance test for a paired difference; report uncertainty on the comparison you actually want to make.

Do not compare absolute ratings across unrelated snapshots as if their origins were fixed. The prompt mix, model versions, opponents, and fitting conventions may differ.

Presentation can affect the outcome

A longer answer may contain more useful detail, or it may merely look thorough. Headings and lists can make an answer easier to use, even when the factual content is unchanged. Whether that should count depends on the task and rubric.

A controlled formatting comparison should preserve substantive content, randomize positions, and collect judgments. If the formatted answer also adds an equation or corrects a mistake, the comparison no longer isolates presentation.

Length-controlled AlpacaEval fits a statistical model of judge preferences using answer-length difference and other features, then predicts preferences with the length difference set to zero. This aims to estimate how the comparison would look at equal lengths under the fitted model.

Arena’s published style-control analysis likewise adds length and markdown features to a preference model. It is a statistical adjustment, not simply an instruction to annotators to ignore formatting. Looking at both adjusted and unadjusted results makes the estimand clearer: ordinary user preference may legitimately include readability, while another analysis may try to isolate different aspects of the responses.

Using a model as the judge

A language model can apply a rubric to many response pairs at lower marginal annotation cost. Its model version, instructions, response order, and treatment of ties all matter.

Zheng et al.’s MT-Bench and Chatbot Arena study examined agreement with human judgments and biases including position and verbosity. Agreement in a particular study does not establish interchangeability with humans on every task. Validate a judge against held-out human ratings for your domain, inspect disagreements, and test position swaps.

Generated responses can also contain text aimed at influencing the judge rather than answering the user. Keep the rubric and quoted responses clearly separated, and test adversarial examples.

Use preference alongside task performance

If a model is preferred but fails more factual or execution checks, inspect the outputs. It might be clearer, more responsive, more detailed, or more persuasive while wrong. The aggregate disagreement does not identify which explanation is true.

A writing assistant needs evidence about reader preference; a coding tool also needs executable correctness checks. Keep the prompt population and cost budget comparable across systems, and use a final holdout after choosing prompts, judges, or model settings.

The last post in this arc asks a related question about individual predictions: when confidence supports an answer, and when to abstain.

Additional reading

• Chiang et al., Chatbot Arena. Pairwise data collection, statistical ranking, and uncertainty.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Human agreement studies and evaluator biases.
• Dubois et al., Length-Controlled AlpacaEval. Regression-based length adjustment and its intended counterfactual.
• Arena, Does Style Matter?. The published adjustment for answer length and markdown features.