Evaluating Reasoning

I've been thinking about what it means to say a model "reasons well," and I keep coming back to this: most benchmarks cited in model releases are measuring something closer to test-taking skill than actual reasoning.

What we care about is whether a model can work through a novel problem step by step, check its own work, and notice when something's gone wrong. What we've been measuring is whether it can pick the right option from a list of four. Those are different.

Harder benchmarks (GPQA, FrontierMath, ARC-AGI) exist as a response to the limitations of MC-based evals. But the saturation problem keeps recurring: a benchmark launches hard, models improve, the benchmark saturates, and a harder successor has to be built. The turnaround from "state of the art benchmark" to "saturated" is now measured in months rather than years.

There's also a second problem running parallel to saturation. Even when a benchmark is hard, multiple-choice format may still be the wrong measurement tool. The same model's apparent capability varies substantially depending on whether it's asked to pick from options or produce an answer. This is the format effect, and it shapes how to read almost any benchmark number you see quoted in a model release.

So this post is about two related things: which benchmarks measure reasoning (saturation and design), and how we measure reasoning even on good benchmarks (format effects and process reward models). Both point at the same issue: a single accuracy number from a multiple-choice test is a poor proxy for what we want to know.

Why we're even here: the multiple-choice problem

The backstory, quickly. Benchmarks like MMLU came out of a reasonable impulse: evaluate language models across a wide range of subjects, systematically, at scale. Multiple-choice is convenient because it's easy to grade. You hand the model four options and check which letter it picked. You can run thousands of questions overnight and get a clean percentage.

The problem is what that percentage is measuring.

A 4-option multiple-choice question has a 25% chance floor. A model that's just guessing randomly scores 25%. A model that can reliably eliminate two obviously wrong distractors (without understanding the content at all) scores 50%. And a lot of MMLU questions have distractors that aren't terribly hard to eliminate. The signal you're left with is squeezed into the top half of the scale.

There's a subtler issue too. Models get very good at recognizing "the shape of a correct MMLU answer." Correct answers in academic multiple-choice have characteristic properties: they tend toward middle values, they tend to be worded more precisely, they tend not to use absolutes like "always" or "never." A model trained on enough MMLU-adjacent text can learn these heuristics without ever learning the underlying content. The score then reflects retrieval and format pattern-matching, not step-by-step reasoning.

By 2024, frontier models were scoring above 90% on MMLU. At that point, MMLU is telling you almost nothing useful about capability differences between models. The benchmark has saturated. You're measuring noise.

Prerequisites

This post builds on the previous Arc 8 posts. You don't strictly need to have read them first, but they set useful context:

• Loss vs. Benchmarks covered why validation loss and benchmark accuracy are two different measurement systems. The format problem in this post is an extension of that idea.
• Eval Harnesses covered how benchmarks get run: prompt formatting, few-shot setup, metric aggregation. The mechanics of how lm-eval-harness handles MC questions matters for understanding the format effect.
• Contamination connects to benchmark saturation and data leakage. A saturated benchmark where training data includes the answers is doubly useless.

You should also know roughly what "chain-of-thought" prompting means: asking a model to show its reasoning steps before producing an answer. We'll use that idea throughout.

Some math: what MC accuracy is actually computing

Let me be specific about what multiple-choice evaluation is measuring, because the probability is worth working through explicitly.

Say a model has access to two different skills: retrieval (does it know the answer?) and format recognition (does it recognize the pattern of correct answers?). On a 4-option MC question, the probability of a correct response is roughly:

If the model knows the answer from retrieval, it gets it right. If it doesn't know, it might still get it right from format heuristics. The observed MC accuracy confounds both signals. A model that knows 60% of the answers by retrieval and has 50% format-heuristic accuracy on the remaining 40% will score around 80% overall, even though it's missing 40% of the underlying knowledge. That 80% number looks impressive but it's obscuring a meaningful gap.

This is why benchmark saturation is such a problem. When a model scores 90%+ on a benchmark, you can't tell how much of that 90% is genuine understanding versus heuristic shortcutting. The ceiling compresses the signal at exactly the moment you'd want it to be most informative. Think of it like a test where every student scores above 95%; the test has stopped ranking anyone meaningfully.

Process reward models take a different approach to the math. Instead of a single accuracy number, they assign a score to each step in a reasoning chain:

where $s_k$ is the $k$-th reasoning step and $r(s_k)$ is the step reward, a value in $[0, 1]$ indicating how likely that step is valid given the steps before it. A chain that arrives at the right answer through wrong reasoning gets a lower total PRM score than a chain that shows clean work throughout. That's the key behavioral difference.

The benchmarks that are still hard

So if MMLU is saturated and GSM8K (grade-school math) is saturated, what's left that's still informative?

GPQA: Graduate-level Google-Proof Questions. Rein et al. built this to be genuinely hard. The questions are written by domain experts (PhDs and graduate students in biology, chemistry, and physics) and vetted by other domain experts. A key design constraint: the questions should be hard even with unrestricted web access. The name "Google-proof" is literal: you can't just search your way to the answer. You need to reason through the domain, integrate several facts, and handle the kind of careful inference that actually distinguishes an expert from a Wikipedia skimmer.

The numbers are interesting. Domain experts with unlimited time and web access score somewhere around 65–75% on GPQA, depending on how strictly you define "expert" and what domains you look at.

Frontier models in 2024 were hitting similar ranges, which makes GPQA a real discriminator. Some models score 45%, some score 70%, and those differences are meaningful because the human baseline is around 65%. A model at 45% is performing well below what a careful human expert would do. A model at 70% is roughly at or above that bar. That's a real signal.

Contrast that with MMLU, where the human baseline is ~89% and frontier models are at 92%, a 3-point gap that tells you almost nothing about the quality difference between systems.

FrontierMath: Competition and Research-Level Math. This one is genuinely brutal. FrontierMath contains problems at the competition math level (IMO-style) and into actual research mathematics, requiring multi-step formal reasoning that can't be shortcut by memorization. The problems are novel, not drawn from existing problem sets, so a model can't solve them by recognizing a familiar template. The baseline performance for frontier models when this benchmark launched in late 2024 was under 2%, i.e. single digits rather than high-80s. (By late 2025 top reasoning models had climbed into the low double digits on the public tier, which is still a dramatic gap relative to MMLU-style saturation.)

That near-zero baseline tells you something important: there's still a massive gap between frontier LLMs and the ability to do original mathematical reasoning. Most existing benchmarks, by showing 85–97% performance, were wildly overstating what models could do on hard novel problems. FrontierMath cuts through that. It's a useful corrective to the general impression that frontier models are near-human on reasoning tasks.

ARC-AGI: Abstraction and Reasoning Corpus. Chollet's argument is that most benchmarks conflate memorization with reasoning. ARC-AGI tries to test pure abstraction: novel visual grid puzzles where the model must induce a rule from a small number of examples and apply it to a new case. There's no training data that can memorize the right answers because the test is specifically designed around novel patterns. You can't have seen these particular puzzles before; you have to actually figure out the rule.

Frontier models have gradually climbed from near zero (2019) to somewhere around 40–50% by mid-2025, which is below human performance (~85%) but not negligible anymore.

Notably, the gains on ARC-AGI came mostly from extended test-time compute strategies rather than raw model size increases. That suggests the bottleneck is something chain-of-thought-style reasoning can partially unlock, not just parametric knowledge. "Think longer" helps here in a way it doesn't help much on saturated benchmarks, where the model already knows the answer or knows the heuristic.

Generation-based evaluation vs. multiple-choice

The deeper issue is the format problem. MC and generation-based evaluation are testing fundamentally different things, even on the same underlying question.

Here's a concrete example. Take an algebra problem, "Solve 2x + 6 = 18." In MC format, the model picks from [A: 4, B: 6, C: 8, D: 12]. In open-ended format, it needs to show the algebra.

The MC version has two pathways to a correct answer: solving it, or recognizing that 6 is a plausible-looking number for this kind of problem without doing any algebra. The open-ended version has only one real pathway: working it through. These are different tasks with different difficulty profiles, and the same model's measured "accuracy" varies substantially between them.

This is sometimes called the format effect, and it's not trivial. Studies comparing MC and open-ended performance on semantically identical questions find model accuracy varies by 10–20 percentage points depending on format. The variation is also sensitive to things you wouldn't expect to matter: option ordering (shuffle A/B/C/D and some models reliably prefer whichever option ended up in position B), option labeling (1/2/3/4 vs. A/B/C/D shifts results), and whether you include "None of the above." These are all irrelevant to the underlying question, but they move the needle.

When a benchmark reports a single number, it's implicitly encoding the format choice. Switch from MC to open-ended and the leaderboard can shuffle noticeably. A model that ranks 3rd on MC format might rank 1st on open-ended, because it's genuinely better at reasoning but less tuned to the particular heuristics that win on multiple-choice.

The reason generation-based evaluation is harder to run is grading. MC is trivially gradeable: compare a letter. Open-ended answers require a judge, either a human or another language model acting as a judge, each with its own biases. This is why MC stuck around long after its limitations were well understood. It was cheap and automatic, and "cheap and automatic" beats "more informative but expensive" in practice for most benchmark pipelines.

There's also a question of what "generation-based" means in practice. If you allow the model to produce a chain-of-thought and then extract an answer, you get one distribution of results. If you constrain it to produce only the answer with no reasoning, you get a different distribution. If you prompt it to "think step by step" vs. just "answer directly," the numbers shift again. None of these are the "true" capability. They're different probes of the same underlying model, each with its own biases. The heterogeneity is real, and it means that quoting a single accuracy number without specifying the exact format is actually pretty underspecified.

How this plays out at runtime

I want to say something concrete about how this affects a real evaluation run, not just the theory.

The lm-eval-harness runs MMLU by default in the standard MC format: prepend a few few-shot examples, append the question, and compare log-probabilities for the four option tokens. The "log-probability" comparison means the model never generates text. It just assigns a probability to each of the four option letters and picks the highest one.

This is fundamentally different from asking the model to produce a solution. The model is being asked "which of these letters is most probable given the context?", not "what is the answer to this question?" In a sense, MC evaluation is running the language model in a very unusual mode, grading on the internal probability of a one-character response rather than the quality of a generated answer. The model's ability to work out the answer is never invoked.

For benchmarks like GPQA and FrontierMath, you can't take this shortcut. The question set is designed so that recognizing "the shape of a correct answer" doesn't help. Grading is necessarily generation-based: produce a solution, then verify correctness (often with another model or a formal verifier for math). The inability to use the MC shortcut is exactly what makes these benchmarks hard to game.

For the reasoning models (o1/o3, Gemini thinking, Claude's extended thinking mode), all evaluations should be generation-based anyway. The core capability being evaluated is the model's ability to produce a long reasoning chain that arrives at a verified answer. Running these models in MC log-probability mode misses the thing they're doing. It's like evaluating a chess engine by having it rate board positions rather than by playing out games.

The practical implication: MMLU and GSM8K scores in a model release are measuring something real (the model knows a lot of things), but the format used to produce them is hiding as much as it's revealing. The numbers worth tracking are GPQA, ARC-AGI, and FrontierMath, because those benchmarks were designed to force the issue.

The saturation lifecycle

The pattern keeps recurring: a new benchmark launches, it's genuinely hard for a year or two, models improve to the point where the benchmark saturates, and then it stops being useful. MMLU launched in 2021. By 2023 frontier models were at 85–88%. By 2024 they were at 90%+. The benchmark went from useful to saturated in about three years.

GSM8K launched in late 2021 as a hard math benchmark. Early GPT-3 struggled. GPT-4 at launch scored 92% (5-shot CoT). By mid-2023 frontier models were at 90%+. Now it's 97%+. That's a useful benchmark's lifespan: about 18–24 months before the most capable models are past the useful ceiling.

Benchmarks that show up prominently in release notes tend to be the ones the model scores well on, which isn't the same set as the most informative measures of capability. Hard benchmarks tend to surface in research papers and appendices rather than in release headlines. That selection effect doesn't require bad faith, just publication pressure and the natural tendency to lead with good results. FrontierMath and ARC-AGI are useful partly as corrective signals: they show the parts of capability space where frontier performance is still limited.

Process reward models: scoring the work, not just the answer

"Let's Verify Step by Step" (Lightman et al., 2023) was the paper that crystallized this for me. The setup: train a reward model to score individual reasoning steps, not just final answers. The key finding: a process reward model (PRM) trained this way was dramatically better at selecting correct solutions from a sampler than an outcome reward model (ORM) trained on final-answer correctness alone.

Why? Because outcome-only scoring has a blind spot. A model can produce a solution that arrives at the correct answer through incorrect reasoning, and ORM misses that. In multi-step math, you can sometimes get the right answer despite an error if errors cancel or if you got lucky guessing a numeric value midway through. ORM gives full credit to the lucky guess, while PRM notices the bad step in the middle of the chain.

In the context of evaluating reasoning, this matters for two reasons. First, it gives you a more fine-grained signal about where a model's reasoning breaks down, not just whether it got the right answer. Second, and more interestingly, PRM scores can be used during generation (not just evaluation) to guide a model toward cleaner reasoning chains via beam search or reranking. This is part of what's happening in o1/o3-style reasoning models, though the details of how OpenAI and others use PRM internally aren't fully public.

The catch is that training a PRM is expensive. You need step-level labels, which means either human annotation at per-step granularity (much more labor-intensive than outcome labels) or using a stronger model to generate the step labels (which can propagate the stronger model's biases). Neither is cheap. That's why PRM is mostly a research-and-frontier tool rather than something you see in commodity eval pipelines.

There's also a practical question about what counts as a "step." In chain-of-thought outputs, the natural boundaries between steps are often fuzzy. The Lightman et al. approach used human annotators to mark step boundaries, but defining a consistent granularity is genuinely hard. Finer granularity gives more diagnostic information but costs proportionally more to label. In production, most implementations use sentence boundaries or paragraph breaks as a proxy, which is imperfect but tractable.

A third use of PRM: best-of-N selection. Rather than scoring a single chain, you can sample N candidate chains and rank them by their PRM aggregate score. The intuition is that a model running unconstrained might produce a mix of correct-reasoning chains and flawed-reasoning chains. If you can score them, you can pick the best one. Lightman et al. showed that this best-of-N-under-PRM strategy was significantly better than best-of-N-under-ORM (picking the chain with the highest outcome probability), because ORM can't distinguish "right answer through correct reasoning" from "right answer through lucky error cancellation."

This is roughly the mechanism behind test-time scaling: running the model multiple times and selecting the best output. PRM is what makes that selection meaningful. Without a reliable step scorer, best-of-N degenerates into random sampling from the model's output distribution, which gains less than you'd hope. With a good PRM, it turns into a search procedure that can reliably find correct reasoning chains even when the model doesn't produce them on the first try.

What "hard to game" actually means in practice

I want to be concrete about what makes GPQA and FrontierMath hard to game, because the property is genuinely interesting.

For a benchmark to resist gaming, it needs to fail the following attack: fine-tune a model on the question distribution until it memorizes the answers. This works for most benchmarks. Train on the MMLU test set (or close approximations) and your model will score 95%+. The questions aren't novel enough to prevent this.

GPQA resists this partly through curation: the questions were written by experts, vetted by other experts for correctness, and then additionally filtered to remove questions that could be Googled. The resulting set is small (448 questions in the main set, 198 in the harder "diamond" subset) but genuinely hard. The expert validation is what prevents the "technically correct but misleadingly phrased" problem that plagues crowd-sourced benchmarks.

FrontierMath resists it differently. The questions are novel, not drawn from existing competition math archives or textbooks. Each problem was constructed by a mathematician specifically for the benchmark. The verification method is computational: the answer is a specific number or mathematical object that can be verified programmatically. So even if a model produces a plausible-sounding explanation, the benchmark can check whether the final answer is actually correct to arbitrary precision. This combination of novelty (can't memorize) and verifiability (can't hallucinate a plausible wrong answer) is what makes the sub-2% frontier baseline so striking.

ARC-AGI uses a different strategy: the test examples are generated from novel rule sets that weren't part of training. You can't memorize your way through it because the grids you see at test time are genuinely new. The flip side is that the evaluation is purely behavioral. There's no chain-of-thought, no explanation, just output grids. That makes it hard to diagnose why a model fails, but very easy to verify whether it succeeded.

These design strategies (expert curation, novel generation, computational verification, behavioral purity) are roughly the toolkit for building benchmarks that stay hard. Each strategy has costs, though. Expert curation is expensive and hard to scale, novel generation requires domain experts to write problems, and computational verification only works for problems with computable answers.

Worked example

Here's how to compare MC-format and open-ended evaluation on the same question, using a language model's log probabilities:

# Comparing MC accuracy vs. open-ended generation on the same question.
# MC: look at the model's log-probabilities for " A", " B", " C", " D"
# Open-ended: generate a solution and check it with a judge or PRM
 
import torch
import torch.nn.functional as F
 
# Simulated logit output from a model given:
# "Question: A store sells 12 apples for $3. How much for 20?
#  (A) $4.50  (B) $5.00  (C) $6.00  (D) $7.20
#  The answer is:"
# Raw logits for the next token (just the option letters here)
logits_for_options = {
    "A": -3.1,   # $4.50
    "B": -1.8,   # $5.00 (correct)
    "C": -2.4,   # $6.00
    "D": -4.2,   # $7.20
}
 
# MC evaluation: pick the highest-probability option
logit_tensor = torch.tensor(list(logits_for_options.values()))
probs = F.softmax(logit_tensor, dim=0)
chosen_idx = probs.argmax().item()
chosen = list(logits_for_options.keys())[chosen_idx]
print(f"MC pick: {chosen} (p={probs[chosen_idx]:.3f})")
# MC pick: B (p=0.571)  <- correct, but we don't know why
 
# Open-ended: score each reasoning step via a PRM
# (In practice you'd call a real model; this shows the structure.)
def mock_prm_score(step: str) -> float:
    """Mock PRM: returns low scores for known error patterns."""
    if "÷ 3" in step:        # wrong divisor, should be ÷ 12
        return 0.08
    if "= $5.00" in step:
        return 0.97
    return 0.92
 
chain = [
    "unit price = $3.00 ÷ 12 = $0.25",
    "20 × $0.25 = $5.00",
    "answer: $5.00",
]
step_scores = [mock_prm_score(s) for s in chain]
prm_aggregate = sum(step_scores) / len(step_scores)
 
print("\nProcess reward scores per step:")
for step, score in zip(chain, step_scores):
    flag = "✗" if score < 0.3 else "✓"
    print(f"  [{flag} {score:.2f}] {step}")
 
print(f"\nPRM aggregate: {prm_aggregate:.3f}")
print(f"ORM (outcome only): {'correct' if chain[-1].endswith('$5.00') else 'wrong'}")
# MC gives you one bit. PRM gives you a diagnostic over the reasoning.

The point is: MC gives you one bit of information per question (right/wrong). PRM gives you a score for every step. On easy questions where everything goes well, the two signals agree. On harder questions where the model gets the right answer through flawed reasoning, they diverge, and the PRM signal is the more informative one.

If you're running your own eval pipeline and want to approximate PRM without training a dedicated reward model, one pragmatic option is to ask a capable judge model to assess each step individually. It's not as well-calibrated as a trained PRM, but it's better than outcome-only scoring, and it's easy to implement with any model that follows instructions reliably.

Misconceptions

"High MMLU scores mean the model reasons well." Not necessarily. MMLU saturates at 90%+, and the portion of questions that actually require multi-step reasoning is pretty small. A lot of MMLU is knowledge retrieval: do you know what the capital of the Weimar Republic was, or what the second law of thermodynamics states? Those are useful things to know, but they're retrieval, not reasoning. A model that scores 88% on MMLU might be far better at knowledge recall than a model that scores 85%, but the 3-point gap tells you almost nothing about reasoning quality on a hard novel problem.

"If the final answer is correct, the reasoning was sound." This is the mistake that outcome-only reward models make, and it happens a lot. In multi-step algebra, canceling errors can produce a correct final value despite wrong intermediate steps. A model that "reasons its way" to a correct conclusion might be following a completely different (and flawed) logical path that happens to land in the right place. Process reward models exist precisely because this phenomenon is common enough to matter for training signal quality.

"GSM8K is still a useful signal." Not anymore for frontier models. With 97%+ accuracy, you're measuring noise. The questions were designed for early-generation language models that were struggling with basic arithmetic. At this point, if a model scores below 90% on GSM8K, that's meaningful. If it scores above 95%, all you know is it's better than the 2021 baseline. The useful ceiling was hit around GPT-4 vintage models. GSM8K remains useful as a sanity check for small or fine-tuned models, but it's not a frontier discriminator.

"Process reward models are just for math." The original PRM paper focused on math reasoning because that's where step-level correctness is easiest to verify. But the conceptual framework applies anywhere you care about the quality of reasoning, not just the correctness of the final answer. There's active work on PRMs for code (where you can run unit tests at each step), scientific reasoning, and argument quality in natural language tasks.

What's next

The next post covers tool and agent evaluation, where evaluation gets significantly harder because you're no longer grading a single answer but assessing a multi-step trajectory. Does partial credit at step 3 count if the agent failed at step 5? How do you score a search query that found the right page but extracted the wrong information? The problems from this post (format effects, outcome vs. process) all show up again, but now layered over an environment that's nondeterministic and where "correct" can be ambiguous in ways that a math answer never is.

Additional reading (and watching)

• Hendrycks, D., et al. (2020). Measuring Massive Multitask Language Understanding. ICLR 2021. The original MMLU paper. Covers 57 subjects, 4-option MC. Became the dominant broad-coverage benchmark until saturation made it less informative for frontier models.

• Lightman, H., et al. (2023). Let's Verify Step by Step. arXiv:2305.20050. The paper that formalized process reward models for math reasoning. Shows that step-level scoring dramatically outperforms outcome-only scoring when selecting among candidate solutions.

• Cobbe, K., et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. The original GSM8K paper. Frontier models now score above 97%, meaning the benchmark has saturated and no longer discriminates between top-tier systems.

• Rein, D., et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022. Expert-written questions in biology, chemistry, physics. Domain experts with web access score 65–75%. Designed to resist surface-heuristic solving and memorization.

• Glazer, E., et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv:2411.04872. Research and competition-level math. Frontier models at launch scored 2–8%. Specifically designed so that memorizing existing solutions cannot crack it.

• Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547. The philosophical and technical foundation for ARC-AGI. Argues that generalization from small examples under a fixed prior is a better measure of intelligence than task-specific performance on familiar distributions.

• Pezeshkpour, P. & Hruschka, E. (2023). Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions. arXiv:2308.11483. Empirical evidence that MC performance is sensitive to option ordering and format, sometimes varying by 10–20 percentage points on semantically identical questions.

• Lightman, H., et al. (2023). Let's Verify Step by Step. Cited again because the PRM vs. ORM comparison is the most direct empirical statement of why scoring reasoning steps differently from final answers matters.

• Phan, L., et al. (2025). Humanity's Last Exam. arXiv:2501.14249. A 2,500-question expert-curated benchmark targeting domains where frontier models score below 10% at time of publication. Designed to remain hard as model capabilities improve.

• OpenAI. (2024). Learning to Reason with LLMs. OpenAI blog post introducing o1. Describes the use of chain-of-thought at test time and process-level reward signals without fully disclosing the training procedure.

• ARC Prize Foundation. (2024). ARC Prize. The $1M competition for reaching 85% on ARC-AGI-2 under constrained compute. Tracks ongoing progress against the novel reasoning frontier and publishes results for submitted solutions.