Loss vs. Benchmarks: Two Languages for Model Quality

Every AI lab announcement follows roughly the same script. A new model drops, there's a table of benchmark scores, and the press release says the model "achieves state-of-the-art" on some set of tasks. Meanwhile, the training engineers who built it have been staring at a different number the whole time: the validation loss.

I ended up thinking about this split more than I expected to, partly because when researchers and practitioners argue past each other about whether a new model is "better," they're often speaking two different languages. One person is talking about loss, the other about benchmarks. Both are valid. They measure different things, and knowing when to reach for which one is the foundation of everything else in this arc.

Why we need two measuring sticks at all

When you train a language model, you're doing one specific thing: trying to make the model assign high probability to the correct next token, given everything before it. The loss function (cross-entropy) measures how well you're doing at that task. After each batch, you check the loss. After each checkpoint, you check it on a held-out validation set that the model hasn't trained on. That's validation loss.

Validation loss has a nice property: it measures exactly the objective you're optimizing. But there's a problem. You're not ultimately trying to build a model that's good at predicting held-out tokens from Common Crawl. You're trying to build a model that's good at helping people answer questions, write code, summarize documents, or reason about math. Token prediction loss is a proxy for those things. A fairly good proxy, as it turns out, but a proxy.

This is why benchmarks exist. A benchmark tries to measure something closer to "is this model actually useful for task X?" It presents the model with a series of questions or problems, records the answers, and scores them. ARC-Challenge asks about science questions a fifth-grader should get right. HellaSwag asks the model to complete sentences in a plausible way. MMLU covers a sweep of academic subjects. The idea is to put the model in front of real tasks and see what happens.

So validation loss and benchmarks are measuring related but different things. Loss asks: how surprised is the model by text? Benchmarks ask: how often is the model right? For most of this arc, we'll be thinking carefully about the gap between those two questions.

What validation loss measures

Let me be precise about what's going on with loss, because a few things about it are easy to misunderstand.

Cross-entropy loss on a held-out validation set is the average, across all tokens in the validation corpus, of $-\log P(\text{correct token})$. The model reads a context, produces a probability distribution over all ~100k tokens in the vocabulary, and we look at the probability it assigned to the actual next token. Then we take the negative log of that probability.

A few things follow from this.

First, loss is smooth and reliable within a training run. Because it's averaging over millions of tokens and is computed exactly from the model's predictions, it doesn't bounce around much. You can see small improvements (a drop from 2.31 to 2.28, say) and trust that they're real. Training curves for loss look like a clean power-law decay toward a floor. That smoothness makes loss a great instrument for debugging training and comparing checkpoints.

Second, loss is not comparable across tokenizers or data distributions. A model trained on a different tokenizer, or evaluated on a different dataset, will have a different loss floor regardless of how "good" it is. You can confidently say "model A has lower loss than model B on the same validation set under the same tokenizer," but comparing raw loss numbers across labs doesn't mean what it looks like it means.

Third, loss is not a direct readout of capabilities. A model can have great loss and still refuse to follow instructions. It can have good loss on general text and poor performance on specialized domains. The gap between "good at predicting tokens" and "good at helping users" is exactly what post-training (fine-tuning, RLHF, DPO) is designed to bridge. We spent all of Arc 7 on that, and the intuition carries here: loss and usefulness are correlated, not synonymous.

What benchmarks measure

A benchmark is an attempt to operationalize "is this model capable of X?" It generally works by presenting the model with a standardized set of prompts, collecting responses, and scoring them against correct answers.

The formats vary. Multiple choice (select A, B, C, or D) is common because it's easy to score automatically. Open-ended generation (write code that passes these test cases) is more realistic but harder to evaluate. Some benchmarks measure short factual recall. Others probe multi-step reasoning. A few try to approximate real user interactions.

The big advantage of benchmarks over loss is that they're interpretable to humans. "64% on ARC-Challenge" means something to a person who has read ARC-Challenge questions. "Validation loss 2.31" means something only if you know the tokenizer, the validation dataset, and the general order of magnitude for models of this type. Benchmark scores travel well in communications because they're attached to specific tasks with human-readable semantics.

But benchmarks have real problems.

They're noisy. Because each benchmark has a finite number of questions (typically a few hundred to a few thousand), and because different random seeds during evaluation can produce different results, benchmark scores have substantial variance. Run the same model twice with different evaluation seeds and you can see swings of several percentage points. The visualization below shows exactly how wide the noise band is, compared to the tight band on loss.

They saturate. When a model gets to 90%+ accuracy on a benchmark, the remaining headroom disappears and the benchmark loses its ability to discriminate between models. MMLU was a meaningful benchmark when GPT-3 scored 43.9%. By 2025, frontier models were scoring 90%+, and the benchmark had basically stopped measuring anything interesting. Every few years the community needs to retire old benchmarks and introduce harder ones, which creates comparability problems.

They're gameable. Because benchmarks are public, training data curation teams know which benchmarks are used for reporting. Some of that data ends up in training sets, intentionally or not. We'll cover this in detail in the contamination post, but the short version is: a high benchmark score from an unknown lab should make you skeptical, not confident.

They measure the wrong thing. Multiple-choice accuracy is a weird format. Real users don't present models with multiple-choice questions. MMLU scores correlate with something about model quality, but the correlation is loose, especially for the kinds of things users care about (is the code correct? is the summary accurate? does the assistant follow the instructions?).

The log-linear relationship with compute

Here's an empirical observation that connects the two measuring sticks: there's a roughly log-linear relationship between compute and benchmark accuracy.

If you plot validation loss against compute (on a log scale), you get a straight line. That's what the Kaplan et al. and Chinchilla scaling laws describe. You can also plot benchmark accuracy against compute (again on a log scale) and see something similar: accuracy tends to improve roughly linearly with log(compute). More precisely, dropping the loss by a fixed amount tends to produce a similar lift in benchmark accuracy, at least while you're in the middle of the scaling curve.

The scatter plot above shows what this looks like empirically. The trend is real: lower loss generally means higher accuracy. But the scatter between different benchmarks is large. Dropping from loss 3.5 to 2.5 might give you +15 percentage points on HellaSwag and only +7pp on TruthfulQA. Different benchmarks capture different things, and the log-linear relationship is a rough average, not a tight per-benchmark law.

This is why scaling law work mostly uses loss as the independent variable rather than benchmark scores. Loss is a cleaner signal. Benchmarks are the outcome you care about, but loss is the instrument you use to steer.

The emergent abilities debate

This is one of the most interesting (and contentious) recent conversations in ML evaluation, and it's directly relevant to how you interpret benchmark scores.

In 2022, Wei et al. published "Emergent Abilities of Large Language Models." They showed plots where a model performs near-chance on some benchmark across a range of model sizes, and then suddenly, at some threshold, performance jumps sharply. They called this "emergence," the idea that qualitatively new capabilities appear discontinuously as models scale, like a phase transition in physics.

This was a striking claim, and the plots were vivid. The story was compelling: small models can't do X, you cross some critical scale, and suddenly they can.

Then in 2023, Schaeffer et al. published "Are Emergent Abilities of Large Language Models a Mirage?" This is where it gets interesting.

Schaeffer's argument was that the "emergence" you see is an artifact of using discontinuous metrics. If you measure a task with exact-match accuracy (a binary 0/1 for each question), then a model that's gradually getting better at the underlying task will look flat until it crosses the 50% threshold on individual questions, at which point accuracy starts rising. The underlying capability is growing smoothly. But the measuring stick is mapping that smooth growth onto a sharp jump.

Switch to a continuous metric, like log-probability of the correct answer, or a partial-credit score, and the phase transition dissolves into a smooth curve.

This is the key visualization for this whole post. Toggle the metric below and watch what happens to the "emergence threshold."

I find this genuinely clarifying. The underlying capability (the model's actual probability of getting the right answer) grows continuously with scale. Binary metrics just can't see the growth until it crosses the decision boundary. The "phase transition" is a measurement artifact.

The debate isn't fully settled. Wei et al. and others have argued that some abilities really are more step-like even under continuous metrics, and that the mirage critique doesn't cover all the cases they documented. But the core epistemological point stands: metric choice shapes what you see. A discontinuous metric will produce phase-transition-looking plots even from smooth underlying dynamics. You should be skeptical of emergence claims that only appear under binary scoring.

Worked example

Suppose you're training a 7B-parameter model and you pull a checkpoint at 100k steps. Same weights, same forward pass. Here's how a training engineer and a release-notes author each read the results:

What do these numbers actually tell you?

The loss says: on average, the model assigns probability $e^{-2.47} \approx 8.5\%$ to the correct next token. Given a vocabulary of ~100k tokens, that's much better than chance (0.001%). Well-trained 7B models tend to approach loss ~2.1–2.2 on typical pretraining distributions, so there's likely room to improve.

The benchmarks say: the model is doing reasonably well on multiple-choice academic tasks, better than random (25% for 4-way MC), and in the ballpark of what you'd expect for a model at this scale.

But here's what the benchmark numbers don't tell you: whether this checkpoint is better than your previous one. A drop from 2.51 to 2.47 in loss is unambiguously an improvement. Moving from 55.7% to 56.2% on ARC-Challenge might just be noise. It's well within the evaluation variance range.

For training decisions (should I keep going? is this hyperparameter working?), loss is the right tool. For communicating about the model to the outside world (is this good enough for our use case?), benchmarks are closer to the right tool.

That's the core of the two-language idea.

How it works in practice

Here's minimal Python to compute validation loss alongside a reminder of how benchmark eval works in practice:

import torch
import math
from transformers import AutoModelForCausalLM, AutoTokenizer
 
# --- Compute validation loss on a held-out corpus ---
def compute_val_loss(model, tokenizer, text: str, device="cuda") -> float:
    """Average cross-entropy (nats) over the tokens in `text`."""
    tokens = tokenizer(text, return_tensors="pt").input_ids.to(device)
    with torch.no_grad():
        out = model(tokens, labels=tokens)
    return out.loss.item()  # already averaged over tokens, in nats
 
# Load a checkpoint
ckpt = "path/to/checkpoint"
tokenizer = AutoTokenizer.from_pretrained(ckpt)
model = AutoModelForCausalLM.from_pretrained(ckpt, torch_dtype=torch.bfloat16)
model.to("cuda").eval()
 
val_text = open("val_slice.txt").read()
loss = compute_val_loss(model, tokenizer, val_text)
perplexity = math.exp(loss)
print(f"val loss: {loss:.4f} nats   perplexity: {perplexity:.1f}")
 
# --- For benchmarks, use lm-evaluation-harness ---
# pip install lm-eval
#
# From the command line:
#   lm_eval --model hf \
#           --model_args pretrained=path/to/checkpoint \
#           --tasks arc_challenge,hellaswag,mmlu \
#           --device cuda \
#           --batch_size 8
#
# The harness handles prompt formatting, few-shot examples,
# normalization, and scoring. Don't roll your own.
# The details matter a lot.
# (Next post covers exactly what lm-eval does under the hood.)
 
# Quick rule of thumb:
#   Loss delta of -0.05 nats  →  training is improving, proceed
#   Bench delta of +1-2pp     →  probably noise, don't report it
#   Bench delta of +5pp+      →  probably real, worth reporting

The out.loss from a HuggingFace causal model is already averaged cross-entropy, using natural log. If you want bits, multiply by $\log_2 e \approx 1.44$.

The comment about lm-eval is genuine advice: don't hand-roll benchmark evaluation. The harness handles few-shot prompt formatting, normalization, and edge cases that matter enormously for getting scores that are comparable to published numbers. We'll go deep on how the harness works in the next post.

Misconceptions

"Lower loss always means a better model." Not across the board. Lower validation loss means the model is better at predicting held-out tokens from the same distribution as training. But cross-tokenizer comparisons are meaningless (a bigger vocabulary might naturally have higher per-token loss because each token covers less text), and models fine-tuned for instruction-following often have higher perplexity on raw web text even though they're more useful. Loss is a proxy. It's a good proxy within a controlled comparison, but it's still a proxy.

"A benchmark score is an objective measurement." Benchmark scores are reproducible if you hold the evaluation setup constant: same harness, same few-shot count, same prompt template, same normalization method. But those choices aren't standardized across labs. A 70% on MMLU from one lab might use 5-shot prompting; another might report 0-shot. The same model can score 10+ percentage points differently just from prompt formatting choices. This is part of why the next post exists.

"Emergent abilities mean there's a minimum scale for usefulness." This was the implicit conclusion some readings of Wei et al. drew: small models fundamentally can't do X, and you need 100B+ parameters to see it. The Schaeffer et al. work suggests this was largely an artifact of binary metrics. Smaller models often have the underlying capability at lower confidence, and whether that lower-confidence version is practically useful depends on the task, not on whether a binary accuracy score crosses some threshold.

What's next

The next post covers eval harnesses: lm-eval-harness and HELM, how standardized evaluation frameworks handle prompt formatting, few-shot setup, and metric aggregation under the hood, and why getting those details right matters a lot.

Additional reading (and watching)

• Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. GPT-3 paper; section 3 describes the training objective (language modeling cross-entropy) and validation setup.

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 3: Probability and Information Theory. MIT Press. Derives cross-entropy from the KL divergence and explains the connection to information-theoretic surprise.

• Hendrycks, D., et al. (2020). Measuring Massive Multitask Language Understanding. ICLR 2021. The original MMLU paper, including GPT-3's initial 43% score and the intended use as a rigorous multi-domain test.

• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv. The foundational scaling law paper: loss scales as a power law of compute, parameters, and data. The loss-centric view of training progress.

• Wei, J., et al. (2022). Emergent Abilities of Large Language Models. arXiv. Documented sharp capability jumps on specific benchmarks as model scale increased, framing them as qualitative phase transitions.

• Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage?. NeurIPS 2023. Shows that emergence is an artifact of nonlinear metrics — switching to continuous metrics reveals smooth underlying capability growth.

• Wei, J., et al. (2022). Inverse Scaling Can Become U-Shaped. arXiv. Documents cases where performance appears to be genuinely non-monotone, suggesting not all surprising scaling behavior is purely metric-driven.

• Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv ("Chinchilla"). Shows Gopher was undertrained and derives the compute-optimal model-size / data-volume ratio. Entire analysis conducted in validation loss space.

• Glazer, E., et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv. A next-generation math benchmark where frontier models scored under 2%, illustrating both the saturation problem with existing benchmarks and the arms race dynamic.

• Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. Introduces the Elo-based Chatbot Arena framework and validates that human pairwise preferences correlate with model quality better than many static benchmarks.