Human Preference Evaluation

I want to start with a confession: I used to look at the Chatbot Arena leaderboard and treat it as ground truth. Model A is at 1247, Model B is at 1218, therefore A is better. Done. The more I dug into how those numbers are produced, though (where they come from, what noise is baked in, what they can't possibly capture), the less clean the story got. That's what this post is about.

Benchmarks measure specific, fixed tasks. They're precise but narrow. Human preference evaluation measures something different: which response do people prefer, in the moment, for the prompt they asked. That's a messier signal, but it's also a much more direct proxy for whether a model is useful in practice. Neither approach is enough on its own.

This post assumes you've read the earlier Arc 8 posts, especially the one on loss vs. benchmarks and contamination. It also uses basic probability vocabulary from Arc 1 (logistic function, expected value). No new math beyond that.

A brief taxonomy of what we're measuring

Benchmark evaluation is mostly measuring capability: can the model do X? Can it solve this math problem? Translate this sentence? Answer this medical question? These are capabilities with (approximately) objective answers. You can grade them automatically. The evaluation is reliable and reproducible.

Human preference evaluation is measuring acceptability: does the model produce outputs that people are satisfied with? This is a much more subjective property, and it's closer to what matters when someone is using a model day to day. You can be capable at something and still produce responses that feel wrong: too formal, too terse, patronizing, hallucinating subtly, refusing when you shouldn't.

There's a third thing, which I'd call safety: does the model avoid producing harmful, dangerous, or policy-violating outputs? Human preference evals mostly don't measure this (users in the Arena aren't asked to assess safety), and neither do most capability benchmarks. That's a whole separate evaluation apparatus (red-teaming, harm classifiers, specific safety evals) which we won't cover here.

Understanding that you're measuring three different things (capability, acceptability, safety) and that no single evaluation captures all three is most of what you need to read model evaluations clearly.

The problem this solves

The thing about traditional benchmarks is that they tell you how a model performs on questions that someone else decided were important. MMLU, GPQA, HumanEval, and the rest are carefully constructed test sets, vetted by humans, designed to measure specific capabilities. That's valuable. But it's also synthetic.

When a user sits down with a model, they're not answering multiple-choice questions about high school chemistry. They're asking it to debug their code, explain a concept, draft an email, work through a problem. The natural distribution of user prompts is completely different from any static benchmark. And the thing users care about (whether the response is helpful) is hard to reduce to accuracy on a fixed answer key.

Human preference evaluation sidesteps this by just... asking humans. You show a person two responses to the same prompt, without telling them which model generated each, and ask: which is better? You aggregate thousands of those votes and you start to get a picture of which models people prefer in practice, across the real distribution of things people ask about.

The leading implementation of this idea is Chatbot Arena, originally from LMSYS and now operating under the LMArena brand. Since its launch in 2023, it's accumulated tens of millions of votes and become probably the most influential model ranking in the field. Frontier labs cite Arena scores in launch announcements. It's the closest thing to a consumer approval rating the industry has.

How Chatbot Arena actually works

The setup is deliberately simple. You visit the site, type a prompt, and get two responses side by side. You don't know which model generated which. You pick the one you prefer, or declare a tie. That's a match. Your vote goes into the system. Repeat at scale.

The "anonymous" part is load-bearing. If you know one response is from a flagship model you've heard good things about, you'll probably rate it higher even if the other response is better. Blinding prevents that. Or at least, limits it. (We'll come back to what it doesn't prevent.)

The pairwise format is also a deliberate choice. Asking "rate this response from 1 to 5" invites anchoring, recency bias, and idiosyncratic scale interpretation. Different people have different internal scales. Asking "which of these two is better" forces a relative judgment, which is much easier to aggregate consistently. It's the same reason chess ranking works pairwise rather than by asking each player to rate themselves.

One underappreciated design decision: the voter's prompt. In Chatbot Arena, the prompt comes from the user; you type whatever you were going to ask anyway. This means the distribution of prompts is the actual distribution of user intent, not a curated benchmark set. That's a significant advantage over any static evaluation. It's also a significant source of noise: the same model pair will get matched on wildly different prompts, and a single vote encodes both the model quality and the prompt-specific context.

The Arena also has a "hard prompt" mode that deliberately filters for more challenging, multi-turn, domain-specific queries where the models meaningfully differ. The default mode includes a lot of short, easy prompts where any competent model will give a good response and the vote ends up measuring stylistic preferences more than capability gaps. Hard prompt mode narrows this, though the user base for hard prompts is smaller.

Votes accumulate. At some point you need to turn a pile of binary match outcomes into a ranking over many models. That's where the Elo system comes in.

Elo and the Bradley-Terry model

The original Elo rating system was invented for chess. The core idea: each player has a latent "skill" number. When two players meet, the probability that the higher-rated one wins is a function of the difference in their ratings.

For LLMs, the same math applies. Each model has a latent skill $r$. When model A and model B are compared, the predicted probability that A wins is:

That's the Bradley-Terry model with a logistic link. The $400$ is a scaling constant (inherited from chess) that makes a 400-point difference correspond to roughly 91% expected win rate. The $\sigma$ is just the sigmoid function.

When a match happens, ratings update. If A wins:

$K$ controls the learning rate. Larger $K$ means ratings move faster (more sensitive to recent matches, noisier). Smaller $K$ means ratings are stable (slower to update, better at long-run averages). Chatbot Arena uses $K \approx 4$ and runs a full maximum-likelihood Bradley-Terry fit rather than the incremental Elo update above, but the intuition is the same: win more than expected, rating goes up; lose more than expected, rating goes down.

Let me work through a concrete example. Suppose model A has rating 1050 and model B has rating 950. The expected probability that A wins is:

So A is expected to win 64% of the time. If A wins, the update is small, since it was expected. If B wins (the upset), the update is large in both directions: B gains more rating, A loses more. The system is self-calibrating: upsets cause big moves, expected outcomes cause small ones. Over many matches, each model's rating settles near the value where it wins as often as expected.

The curve below is worth staring at for a second. It's the Bradley-Terry link between an Elo gap and a win probability, and the thing that surprised me the first time I actually plotted it is how sigmoid-shaped the relationship is near zero. A 50-point Elo gap only corresponds to a 57% win rate. A 100-point gap gets you to 64%. You need to open a 200-point gap before the stronger model is winning three times as often as losing. That's the conversion rate between "Elo points" and "actual preference probability," and knowing it is a prerequisite for reading leaderboards sensibly.

A few things to notice in that simulation. Convergence takes a long time. With five models and 1000 matches, you're averaging 400 matches per model, but those are spread unevenly because the pairing is random. The confidence intervals matter too: a 20-point Elo gap, even with 10,000 votes, can easily be statistical noise. A 100-point gap is meaningful. A 10-point gap is almost never.

This is the part that gets underreported when Arena numbers show up in launch announcements. A model jumping from 1218 to 1231 isn't necessarily a real improvement. It might be noise. The Arena team does publish confidence intervals, and they're wide.

Inter-annotator agreement: humans disagree

Before we go further, there's a ceiling we need to acknowledge. Humans don't agree with each other about which LLM response is better, and that disagreement isn't noise around a true answer. It reflects genuine variation in what different people value.

Research on preference labeling consistently finds that annotators agree on roughly 70% of pairs. That means for three in ten comparisons, two reasonable humans would pick different responses as better. The sources of disagreement include: different priorities (precision vs. brevity vs. warmth), different backgrounds (an expert might prefer a technically correct but terse response while a non-expert prefers a longer, more scaffolded one), and genuine ambiguity in what "better" even means for a given prompt.

This isn't the same as inter-rater reliability on a factual question, where one person is right and the other is wrong. When someone asks "write me a poem about autumn," there's no objective correct response, and the preferences of different readers are legitimately different. Preference evaluation is measuring aggregate taste over a diverse population, not correctness.

This 30% disagreement rate is a fundamental ceiling on any preference-based evaluation system. It doesn't mean the signal is useless. It means you need a lot of votes before differences become meaningful. It also means that a model reliably preferred by 55% of users (which corresponds to roughly a 35 Elo point advantage over a 50/50 baseline) is genuinely better across the population. But a model preferred 52% of the time might just be fluctuating around the ceiling.

The practical implication: when two models are within 20-30 Elo points of each other, you should look at them as essentially equivalent for most purposes. Treating those gaps as decisive is over-reading the precision of the instrument.

The biases that mess everything up

The preference signal is noisy, but the bigger issue is that it's systematically biased. Humans making pairwise judgments under time pressure exhibit predictable patterns that have nothing to do with response quality.

Length bias. Longer responses win more often. Studies on LMSYS data found that response length is one of the strongest predictors of Arena win rate, stronger than most quality signals. A response that says the same thing in 300 words instead of 100 words tends to be rated higher, even if the additional 200 words don't add any information. This is partly because length reads as effort ("they really tried to answer my question") and partly because there's a semantic halo where verbosity feels authoritative.

Markdown and format bias. Bullet points, bold headers, and numbered lists consistently win against plain prose with the same content. Structured formatting reads as cleaner and more professional, and voters reward it even when a narrative paragraph would actually communicate the answer better. There are prompts where a bulleted list is actively the wrong format (narrative questions, nuanced tradeoffs, emotional support) and formatted responses still win on them.

Eloquent-but-wrong bias. This one is subtle. Confident, well-written responses that contain factual errors can defeat accurate but clumsier responses in head-to-head votes. Voters, especially non-expert voters, can't always verify the facts. What they can assess is fluency, structure, and tone. A response that reads well and sounds authoritative will often beat a response that is correct but flat.

These biases aren't unique to crowdsourced annotation. They show up in expert annotation too, just to a lesser degree.

Controlling for style: LC-AlpacaEval

If you already know that length inflates win rates, you can try to remove that confound explicitly. That's what Length-Controlled AlpacaEval (LC-AlpacaEval) does.

The core idea is a regression adjustment. You fit a model predicting win rate from a combination of quality signals and length. Then you report the quality-adjusted win rate at a fixed length baseline, typically normalized to match the reference model's average length. If Model A wins 60% of the time but generates responses that are 40% longer than the baseline, its LC win rate adjusts down to reflect how it would do if lengths were matched.

In practice, you run 805 instruction-following prompts (the AlpacaEval test set), generate responses from your model and from the reference model (typically GPT-4 Turbo), and use an annotator model to judge each pair. The LC adjustment fits a logistic regression on (quality signal, response length) → win outcome and reports the marginal quality effect.

LC-AlpacaEval correlates better with GPT-4 quality judgments than raw win rate, and it's more robust to models that "game" the length dimension. It's not a complete solution (you can't regression-out every stylistic confound simultaneously) but it's a meaningful step toward isolating the signal from the noise. One limitation: the AlpacaEval test set is still a fixed set of prompts, which means models can (and do) optimize directly for it over time. Length-controlling for a gameable metric doesn't fix the underlying distribution shift problem.

LLM-as-a-judge: scaling preference evaluation cheaply

Crowdsourced human evaluation is expensive and slow. You need to wait for real users to visit the site, answer the right kinds of prompts, and vote. This makes it impractical to evaluate models daily during development.

The solution the field has largely converged on is LLM-as-a-judge: use a strong model (typically GPT-4 or Claude) to evaluate response quality instead of human annotators. The evaluator receives the prompt, both responses, and a structured rubric. It produces a preference judgment.

Zheng et al. showed that GPT-4 as a judge agrees with human preferences roughly 80% of the time, comparable to the inter-human agreement rate of ~70%. This is a striking result. A model that agrees with humans 80% of the time is about as good a judge as another human. And it's orders of magnitude cheaper to run at scale.

LLM-as-a-judge is now the standard in automated preference evaluation, powering systems like AlpacaEval (which uses GPT-4 Turbo or Claude as evaluator).

A few caveats: LLM-as-a-judge inherits the biases of the judge model, so if your judge prefers longer responses (which GPT-4 has been observed to do), your evaluation metric will favor verbose models even when controlling for this is the whole point. Using the same model to evaluate its own outputs (or outputs from the same family) also introduces a conflict of interest that's hard to fully eliminate. And the judge's agreement rate with humans is typically measured on a generic distribution; on narrow technical domains where the judge model is itself uncertain, that agreement rate degrades.

None of this makes LLM-as-a-judge useless. It's cheap, genuinely useful, and better than no preference evaluation at all. But it should be calibrated against at least some gold-standard human annotation before you treat its outputs as ground truth.

Here's what the actual Elo update loop looks like when you strip away the dressing:

import math
from collections import defaultdict
 
# Minimal Elo / Bradley-Terry update for a preference tournament.
# Ratings start at 1000. K controls the learning rate.
 
def elo_expected(rating_a: float, rating_b: float) -> float:
    """P(A beats B) under Bradley-Terry with scale factor 400."""
    return 1.0 / (1.0 + 10 ** ((rating_b - rating_a) / 400))
 
def elo_update(
    ratings: dict[str, float],
    winner: str,
    loser: str,
    K: float = 32.0,
) -> None:
    """Update ratings in-place after a single match."""
    rW, rL = ratings[winner], ratings[loser]
    e_win = elo_expected(rW, rL)
    e_los = 1.0 - e_win
    ratings[winner] += K * (1.0 - e_win)   # won more than expected? small bump
    ratings[loser]  += K * (0.0 - e_los)   # won less than expected? small drop
 
# --- Run a small tournament ---
models = ["Apex", "Blaze", "Crest"]
ratings = defaultdict(lambda: 1000.0)
 
# Simulated match results (winner, loser)
matches = [
    ("Apex", "Blaze"),
    ("Apex", "Crest"),
    ("Blaze", "Crest"),
    ("Apex", "Blaze"),
    ("Crest", "Blaze"),   # upset
]
 
for winner, loser in matches:
    elo_update(ratings, winner, loser)
    print(f"{winner} beat {loser}  →  {winner}: {ratings[winner]:.0f},  {loser}: {ratings[loser]:.0f}")
 
# Apex:  1057  (won most)
# Blaze:  979  (mixed record)
# Crest:  963  (won one, lost two)

That's the full Elo update loop. The math is simple. What makes it useful is the scale: tens of millions of matches, spread across a real distribution of user prompts, with blind pairwise judgments.

Arena and benchmarks: why you need both

I keep framing this as "benchmarks vs. human preference" as if you have to choose. You don't, and picking one over the other misses half the picture.

The complementarity goes like this. Automated benchmarks (MMLU, GPQA, HumanEval, MATH) are good at measuring specific capabilities with high precision and low noise. A model either gets the chemistry question right or it doesn't. The score is reproducible, unaffected by stylistic choices, and captures a real dimension of capability. What it can't capture is whether users will find the model's responses useful across the much wider space of natural language interactions, meaning the things people actually ask about, in the registers they actually use.

Human preference evaluation fills that gap. It measures something closer to "will this response satisfy the person asking?" across the real distribution. It's noisy, biased, and gameable, but it's measuring something that benchmarks structurally can't: aggregate usefulness in the wild.

The failure modes go in opposite directions, which is why you want both. A model can score high on MMLU by correctly reasoning through multiple-choice questions while generating responses that users find abrasive, verbose, or oddly formatted. A model can score high on Arena by generating fluent, beautifully structured responses that occasionally hallucinate facts that users can't verify. Neither type of error shows up clearly in the other evaluation regime.

This complementarity has a practical interpretation for practitioners: if you're choosing between two models for a production application and they're similar on both Arena and benchmark scores, that's reasonably strong evidence they're genuinely close. If they diverge (one scores higher on Arena but lower on domain benchmarks), you need to think carefully about which failure mode matters more for your use case.

Can models game the arena?

Yes, and some have.

The gaming surface is everything we already identified as a bias: length, formatting, confident tone, bullet points. A model that is deliberately verbose, heavily structured, and states every answer with high confidence will tend to win more Arena matches than a model with equivalent (or superior) substantive quality but plainer style. This isn't a hypothetical. After the Arena dynamics became widely understood, there was a period where several frontier models were clearly optimized for Arena-friendly presentation patterns in their post-training.

The LMSYS team has explicitly noted that some models appear "Arena-optimized" and that the length-and-formatting effect is large enough to move hundreds of Elo points without necessarily corresponding to meaningful capability improvements.

This is the classic Goodhart's Law problem: when a measure becomes a target, it ceases to be a good measure. Once Arena was the de facto industry leaderboard, it attracted optimization pressure, and the measure started to drift.

Style Control, a variant where annotators are explicitly asked to ignore formatting and length, reduces this gaming surface. But it requires deliberate effort from annotators, and even then style influences perception in ways that are hard to consciously override.

The deeper problem is that the "gaming" behaviors aren't entirely separable from genuine quality improvements. A well-organized response with headers and bullet points often is easier to read and more useful. Longer responses often do contain more relevant information. Confident assertions are often correct. The signal is corrupted, but the corruption isn't random. It's noise in a direction that loosely correlates with quality, which is part of what makes it hard to fully factor out.

What you can say fairly clearly is this: if a model's Arena rank jumps 50 points between releases while the length and markdown subscores go up substantially and the factual accuracy subscores don't move, most of that gap is explained by stylistic variables rather than capability. If length and format stay flat and the hard-prompt category scores go up, the gap is more likely a real capability improvement. Looking at Arena scores across categories and controlling for the stylistic variables gives you a cleaner read than raw rank.

Confidence intervals and what they actually mean

I want to spend a moment on the statistics, because this is where a lot of Arena number-reading goes wrong.

The standard error on a Bradley-Terry estimate scales roughly as $1/\sqrt{N}$ where $N$ is the number of relevant matches for a model. With 500 votes, the 95% confidence interval is wide (probably ±50 Elo points or more). With 50,000 votes, it's maybe ±5 points. Most leaderboard comparisons involve models that are within 20 points of each other. That's often not statistically distinguishable.

The Chatbot Arena leaderboard does display confidence intervals if you look for them. They're easy to miss when you're scrolling a sorted list, which makes the ordering feel more like ground truth than it is. The ordering is a point estimate with wide error bars, for models that are genuinely close together in quality. Treating rank 7 vs. rank 12 as a meaningful distinction, when neither the underlying matches nor the variance estimates support that confidence, is over-reading the instrument.

To be concrete: if Model A has 1240 Elo and Model B has 1232 Elo, and both have ~10,000 votes, those are statistically indistinguishable. A launch announcement claiming "beats Model B by 8 Arena points" is noise, not signal.

There's a pattern worth recognizing in how Arena numbers get used publicly. When a lab releases a model, they often wait until it has accumulated enough votes to show a favorable number, then announce. The announcement freezes the rating at a propitious moment. But Arena ratings drift over time as the comparison pool changes. New, stronger models enter the Arena, and existing models' ratings get recalculated against a harder field. A model that launches at 1280 might stabilize at 1240 once it's been matched against everything. The announcement number is often the high-water mark, not the equilibrium.

This is worth knowing when you read launch posts and see Arena scores prominently featured.

Misconceptions

"Arena rank is an objective measure of model quality." Arena rank is an aggregate of user preferences, measured under specific conditions (blind, pairwise, self-selected user population, real distribution of prompts). That's valuable, but it's still a proxy, and a proxy with known biases, a meaningful noise floor, and susceptibility to gaming. I treat it as evidence rather than verdict.

"A higher Arena score means the model is more helpful." "Helpful" has many definitions, and Arena's pairwise format captures one of them: immediate perceived quality. It systematically underweights things users can't evaluate in real time (factual accuracy, safety properties, consistent behavior across edge cases). A model could score 1300 on Arena and still hallucinate confidently on 15% of factual queries. The users voting probably can't tell.

"Human preference is a ceiling on AI quality." The 30% inter-annotator disagreement is a ceiling on the resolution of the preference signal, not on model quality. A model can be genuinely, verifiably better at a task while losing preference votes because its correct answer is expressed in an unfamiliar register. Human preference is one measurement instrument. It has a limited dynamic range for discriminating between high-quality models.

"More votes always means better estimates." More votes reduce variance, but they can't remove bias. If the annotator pool is systematically skewed toward users who prefer verbose formatted responses, more votes from that pool will just give you a more precise estimate of a biased quantity. Volume helps noise; it doesn't help systematic error.

What's next

Preference evaluation tells you which outputs users like. It doesn't tell you how confident a model is in those outputs, or whether that confidence is calibrated to its actual accuracy. A model can score 1300 Elo and still be wildly overconfident on the questions it answers wrong.

The next post covers calibration and abstention: whether a model's stated confidence matches its actual accuracy, and why teaching models to say "I don't know" turns out to be harder than it sounds.

Additional reading (and watching)

• Chiang, W.-L., et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132. The original paper describing the Arena platform, its Elo methodology, and early results across 20+ models.

• Elo, A. E. (1978). The Rating of Chessplayers, Past and Present. Arco Publishing. The original formulation of the Elo rating system, developed for chess and now applied to everything from tennis to LLMs.

• Bradley, R. A. & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3–4), 324–345. The statistical model underlying Arena's maximum-likelihood rating — every "which model do you prefer?" comparison is a Bradley-Terry observation.

• Boubdir, M., et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295. Analysis of Elo's statistical properties when applied to LLM evaluation, including the gap between incremental updates and full MLE.

• Karpinska, M., Akoury, N., & Iyyer, M. (2021). The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation. EMNLP 2021. Careful study of inter-annotator agreement in text quality evaluation, documenting the ~30% disagreement rate and its sources.

• Dubois, Y., et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475. Quantifies the length bias in Arena-style win rates and introduces the regression-correction approach.

• Dubois, Y., et al. (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. NeurIPS 2023. The original AlpacaEval framework, showing that LLM-based auto-evaluators can substitute for human raters at lower cost.

• Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. The paper that established LLM-as-a-judge as viable — GPT-4 as judge agrees with human annotators at rates comparable to human-human agreement.