Arc 8: Evaluation & Scientific Discipline

Eval Harnesses: lm-eval-harness and HELM

5 min readbuilds on Loss vs. Benchmarks: Two Languages for Model Quality and Cross-Entropy, KL Divergence, and What Loss Functions Measure

What belongs beside a score?

Input: Dataset revision, split, selected items, and few-shot examples.
Model request: Checkpoint and tokenizer revisions, exact rendered prompt, backend, decoding settings, and budget.
Scoring: Candidate strings or answer extractor, normalization rule, metric, and aggregation.
Output: Per-item predictions and errors, aggregate score, uncertainty, and harness version.

Two evaluations of the same checkpoint can disagree because they asked different questions. One supplied five examples; another supplied none. One ranked answer strings; another generated an explanation and extracted a letter. Both may report “accuracy,” but that label does not make their procedures equivalent.

An evaluation harness makes the procedure explicit and repeatable. It loads items, renders prompts, calls a model, scores outputs, and aggregates results.

This builds on loss versus benchmarks and cross-entropy. We will use conditional log-probabilities to score candidate answers.

From a dataset item to a score

From a test item to a reported result

1. Select: Load a pinned dataset split; keep evaluation items out of few-shot selection.
2. Render: Combine instructions, examples, answer options, and the model’s chat template when appropriate.
3. Request: Score candidate continuations or generate a response under a recorded budget.
4. Judge: Apply the specified normalization, parser, exact matcher, or other evaluator.
5. Aggregate: Save per-item results before computing the summary and uncertainty.

A task definition specifies the dataset and split, the prompt format, the allowed answers or generation settings, and the metric. Few-shot examples need their own selection rule. If an evaluation item appears among its own demonstrations, the result is no longer a held-out test of that item.

Save the rendered requests and per-item responses. A configuration file can look correct while the actual prompt duplicates a chat delimiter, loses an option through truncation, or presents an answer in the wrong place. Inspecting a few requests catches failures that an aggregate number conceals.

Record the model and tokenizer revisions, backend, precision, batch configuration, and seeds. Changes in services or numerical implementations can affect outputs even with temperature-zero generation.

Candidate likelihood and generated answers

For a candidate continuation $a=(a_1,\ldots,a_m)$ after prompt $q$ , a likelihood-based evaluator can compute

s(a\mid q)=\sum_{t=1}^{m}\log p(a_t\mid q,a_{<t}).

It selects the candidate with the largest score. Depending on the task, the candidates might be full answer strings or the letters A, B, C, and D after a prompt that already contains the options. Those are different conditional events. A multi-token answer needs the whole continuation scored, not just its first token.

Raw sequence likelihood tends to penalize additional tokens. Some tasks therefore use a normalization rule, but “normalized accuracy” is not one universal operation: inspect the task and implementation to see whether it divides by character count, token count, or another quantity. Changing that rule can change the winning answer.

Generation-based evaluation instead requests a response under a decoding policy and budget, then judges what came back. An exact matcher might accept a single answer string. A parser might extract a number or letter. Executable tasks may run tests. The parser is part of the measurement, including its handling of ambiguous or malformed answers.

Scoring candidates and parsing a response

Authored example: the capital of France. Likelihoods below are assumed, not model measurements.

Candidate log-likelihoods: Paris −0.12; London −3.41; Berlin −4.02; Madrid −3.89. Maximum-likelihood choice: Paris.

Generated response

The answer is (A) Paris.

Parser: first parenthesized A–D letter. Extracted: A. Score: correct.

A parser can miss a correct answer or extract an abandoned choice. Inspect its behavior on actual responses; a likelihood ranking answers a different question.

The generated responses are authored examples. Try the response that mentions an abandoned option: a first-match parser extracts the wrong letter.

Likelihood scoring can batch or cache candidate requests; it does not necessarily require a separate full forward pass per option. Generation also tests whether the model can produce an answer without supplied candidates.

Few-shot examples and templates

Few-shot examples can communicate an output format, teach a mapping, or suggest a procedure. They also change the context and consume its budget. Their effect depends on the examples, their order, the task, and the model. More examples are not guaranteed to help.

Few-shot examples become part of the input

Authored arithmetic examples, separate from the evaluation question.

Examples: 1

Give only the integer answer.

Q: 2 + 2?
A: 4

Q: 6 + 7?
A:

A template should match the model interface and the task. A base completion model and an instruction-tuned chat model may need different wrappers. For a controlled comparison, document that adaptation rather than silently treating all wrappers as interchangeable.

A template defines the requested task

Authored prompt variants.

Format

The capital of France is

The first two forms expect answer text. The third asks for a letter. Changing the prompt without matching the scored continuations or answer parser changes the evaluation.

Prompt selection itself can overfit a benchmark. If you try twenty templates on the final test set and report the best, the score includes that selection process. Choose templates on development data and reserve a separate set for the reported comparison.

Using lm-evaluation-harness

EleutherAI’s LM Evaluation Harness supplies task definitions, model adapters, and scoring machinery. Pin the package or repository revision together with the model and dataset revisions. A task name alone does not freeze its implementation forever.

The current documented command interface uses lm-eval run. Older releases commonly use lm_eval without that subcommand; follow the documentation for the version you install.

The backend has to support the requests a task needs. A chat generation endpoint is not automatically a candidate-log-likelihood endpoint. Check adapter support before interpreting a missing metric or attempting to substitute generated text for unavailable probabilities.

Task YAML makes the intended behavior reviewable. This small custom task expects a local JSON Lines file containing question, choices, and a zero-based integer answer. It evaluates full option continuations with raw likelihood and uses no few-shot examples.

Keep this file alongside the data revision and rendered samples. The harness documentation describes how to register custom tasks and add generation filters, multiple references, or a different aggregation.

HELM and multiple measurements

Stanford’s HELM takes a broader approach to evaluation: define scenarios and report several relevant metrics under standardized conditions. The original work included accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency where applicable. Those dimensions can reveal tradeoffs that one leaderboard column hides.

HELM is a family of evolving evaluations, so name the suite and version used. Do not assume that a result from one suite or adaptation can be compared directly with a superficially similar task elsewhere.

Reproducing a result

When two reported scores disagree

Compare the configurations before attributing the difference to the model.

Same data?: Revision, split, item count, and contamination filtering.
Same requests?: Few-shot examples, template, model revision, context truncation, and generation budget.
Same scoring?: Raw likelihood versus normalized likelihood; parser, reference answers, ties, and invalid outputs.
Same execution?: Backend, precision, batch settings, random seeds, and any provider changes.

Begin with the exact checkpoint and reported configuration. Compare rendered prompts before spending a large inference budget. Check item counts, candidate strings, and answer extraction. Then compare per-item outputs; a batch of invalid generations suggests a different problem from a small set of genuine answer disagreements.

When differences remain, report them. A provider revision, unavailable training checkpoint, or omitted prompt template can prevent exact reproduction. That limitation belongs beside the score, rather than being concealed by rounding both results to the same percentage.

The next problem remains even with a perfectly reproduced procedure: did the model encounter the test material during training? That is the subject of contamination.

Additional reading

EleutherAI, LM Evaluation Harness. Source, supported backends, and installation instructions.
Harness command interface. The command syntax used above.
Harness task guide. Dataset splits, prompt fields, candidate scoring, and metrics.
Liang et al., Holistic Evaluation of Language Models. Scenarios, multiple metrics, and standardized evaluation.