Loss vs. Benchmarks: Two Languages for Model Quality

A model can improve its predictions without answering one more benchmark question correctly. It can also gain a point on a benchmark while getting worse at something the test never asks.

Validation loss measures the probabilities assigned to held-out tokens. A benchmark measures performance under a particular set of questions, prompts, and scoring rules.

What validation loss measures

In cross-entropy and loss, we treated a correct token’s negative log-probability as its surprise. Average that quantity over the scored tokens in a validation set:

Natural logarithms give nats per token. Perplexity is $e^L$. The model can improve this loss by putting more probability on observed tokens even if its most likely next token never changes.

For instance, moving the correct option’s probability from 0.60 to 0.80 reduces its loss from about 0.511 to 0.223 nats. If that option was already the most likely, a greedy answer stays the same. Conversely, a tiny probability change can swap the top two options and flip a correctness indicator.

Sampling noise, optimization dynamics, distribution differences, and overfitting can move validation loss upward between checkpoints.

To compare two losses, hold the validation corpus, tokenizer, context handling, and scored positions constant. A token is not a fixed amount of text across tokenizers. Bits per byte can help compare likelihoods on the same underlying text, but the text, encoding, and evaluation protocol still need to match.

What benchmarks measure

A multiple-choice benchmark might score whether the highest-likelihood candidate matches the answer key. A code benchmark might execute tests. A conversational evaluation might collect preferences. These are different observations, each defined partly by the evaluator.

Prompt wording, few-shot examples, answer normalization, and decoding budget belong to that definition. A score obtained by ranking four supplied answers cannot be substituted for a score obtained by generating an answer without those choices. The next post follows those details through an evaluation harness.

Evaluation uncertainty can come from item sampling, stochastic responses, few-shot selection, nondeterministic execution, or changes to the service. State which source an interval describes. Repeating a deterministic run on the same fixed items adds no observations of item-sampling variation.

For independent binary outcomes with common success probability $p$, the standard error of an accuracy estimate is approximately $\sqrt{p(1-p)/n}$. Questions within a subject or several attempts on one task can be correlated, so that expression is not a universal uncertainty calculator. Comparing models on the same items also calls for a paired analysis, which uses the disagreements between their predictions.

Assess a score difference using the sample size, dependence between items, comparison design, and practical effect. Near-perfect scores can still detect regressions, though a test with few remaining errors has limited room to distinguish improvements.

Loss, compute, and downstream performance

Scaling-law studies fit regularities in loss over specified model families and training regimes. A common illustrative form is

Here $C$ is compute and $L_\infty$ is a fitted asymptote. Under this formula, $\log(L-L_\infty)$ is linear in $\log C$, so the straight-line relationship applies to loss above the asymptote.

Kaplan et al. and Hoffmann et al. study how training resources relate to language-model loss and downstream results under their experimental conditions. Their fits are useful for planning within a regime; they do not give a universal conversion from nats to task competence.

Why a smooth improvement can look abrupt

The emergence debate concerns cases where a measured ability seems to appear suddenly with scale. Wei et al. documented such patterns. Schaeffer et al. showed that nonlinear metrics and limited test resolution can produce apparent discontinuities from smoothly changing underlying performance.

Consider a constructed task with $k$ independent parts, each solved with probability $p$. Expected partial credit is $p$, while the probability of getting the entire task right is $p^k$. With eight parts and $p=0.5$, complete success is only $1/256$. A small test may observe no successes. At $p=0.8$, complete success is about 16.8%. Nothing discontinuous happened to $p$.

A sampled single answer with correctness probability $p$ has expected accuracy $p$, not a threshold at 50%. Greedy decoding does introduce decision boundaries when the top candidate changes, but with several options that boundary need not occur at 50% probability.

A small calculation you can check

If validation loss falls from 2.30 to 2.25 nats, perplexity falls from about 9.97 to 9.49. The geometric mean probability assigned to observed tokens rises from $e^{-2.30}\approx0.1003$ to $e^{-2.25}\approx0.1054$.

That is a roughly 5.1% relative increase in the geometric mean, not the arithmetic mean token probability and not a 5.1-point accuracy gain.

Run in evaluation mode, exclude padding and any unscored targets, sum losses over scored tokens, and divide by their total count rather than averaging unequal batch means. Respect the model’s context limit. Sliding-window evaluation can give later tokens more preceding context than disjoint chunks; record the window and stride so another run scores the same conditional predictions.

For task evaluations, save per-item outputs as well as summary scores. Those outputs let you inspect regressions, test the answer parser, and make paired comparisons. Keep a held-out test set for the final comparison after using development data to choose prompts or checkpoints.

What to carry into an evaluation report

Report the loss protocol, task protocol, and model revision. Show uncertainty appropriate to the sampling process. Distinguish a better probability model on the validation distribution from a better system on the task you care about. When the measurements disagree, inspect the examples before deciding which one to discard.

The eval harnesses post turns that report into a reproducible procedure.

Additional reading

• Kaplan et al., Scaling Laws for Neural Language Models. Empirical loss scaling within the paper’s training regimes.
• Hoffmann et al., Training Compute-Optimal Large Language Models. Allocation of compute between parameters and training tokens.
• Wei et al., Emergent Abilities of Large Language Models. The emergence observations and framing.
• Schaeffer et al., Are Emergent Abilities of Large Language Models a Mirage?. How metric choice and test resolution can create apparent emergence.
• Hugging Face, Perplexity of fixed-length models. Context windows, sliding evaluation, and token-weighted likelihood.