Eval Harnesses: lm-eval-harness and HELM

You've probably seen a headline like "Model X achieves 86% on MMLU." That headline number is the output of about a dozen decisions that don't appear in the headline itself.

This post is about eval harnesses: the layer of software that sits between a raw model and a published benchmark number. The harness decides how to format the prompt, how many examples to include, how to extract an answer from the model's output, and how to aggregate everything into a single percentage. Each of those decisions can swing the final number by several points, sometimes by more than ten.

It's easy to have a vague sense that "eval is complicated" without having worked through exactly how it's complicated. Let me try to build the intuition from scratch.

Prerequisites

This post sits in Arc 8 (Evaluation & Scientific Discipline). Before reading it, you want a working mental model of next-token prediction, sampling strategies, and the idea that a model assigns probabilities to sequences, which we covered in probability distributions. You don't need anything from Arc 8.1 (loss vs. benchmarks) to follow along, but that post sets up the motivation for why we even care about benchmarks beyond training loss.

What a harness actually does

A benchmark is a dataset: a bunch of questions with known right answers. A harness is the scaffolding that operationalizes that dataset for a specific model.

A harness does four things:

Prompt formatting. It takes the raw question and wraps it in whatever text structure the benchmark calls for. That might be as simple as pasting the question verbatim, or as involved as prepending five carefully selected examples with answers, wrapping everything in a chat template, and appending "Answer:" at the end.

Few-shot example management. Most benchmarks are run in a few-shot setting (commonly 5-shot). The harness picks which examples to prepend, from where in the dataset, and in what order. These choices affect scores.

Answer extraction. After the model produces output, the harness has to figure out what the model "said." For multiple-choice benchmarks there are two very different approaches, and I want to spend real time on both of them.

Metric computation. Finally the harness aggregates individual question results into an accuracy percentage, weighted mean, or whatever the benchmark specifies.

The thing I want you to walk away with is that none of these steps are neutral. Every one of them is a modeling choice, and different choices produce different scores on the same model and the same questions.

Two ways to score a multiple-choice question

There are two very different ways to ask a language model a multiple-choice question, and they produce different numbers.

Loglikelihood scoring never generates anything. Instead, for each answer option (A, B, C, D), it concatenates the option text to the question stem and asks: what is the log probability the model assigns to this particular continuation? The model processes each stem-plus-option pair and returns a scalar. The option with the highest log-probability is the model's "answer."

Concretely, for a question "The capital of France is" with options A=Paris, B=London, C=Berlin, D=Madrid, the harness computes:

for each option $o \in \{A, B, C, D\}$, and returns $\arg\max_o \text{score}(o)$. No sampling, no temperature, no generation budget. Just four forward passes and a comparison.

Generate-and-extract works the other way. The harness formats the prompt (usually ending in something like "Answer:"), samples from the model until it produces a stopping token, and then runs a regex over the generated text to pull out an answer letter. If the regex matches "A" or "(A)" or "Answer: A" somewhere in the output, that's the prediction.

Both methods have the right answer. They often disagree with each other.

The loglikelihood method is faster, and it doesn't actually require the model to "understand" the question in the usual sense. It just has to rank the four continuations. The generation method is closer to how a model is used in practice (you'd ask it a question and read its response), but it introduces extraction failures: cases where the model technically gives the right reasoning, but outputs the letter in a format the regex doesn't match.

Which one should you use? The honest answer is: it depends on what you're trying to measure, and the field hasn't fully converged on a standard.

lm-eval-harness: the community standard

EleutherAI's lm-eval-harness is the closest thing to a community standard that exists for open-weight model evaluation. It's what the Open LLM Leaderboard uses (or used to use, more on that later). It supports 200+ tasks, handles both loglikelihood and generation-based scoring, and is extensible via YAML task configs.

The basic mental model of how it's organized:

Tasks are the unit of evaluation. Each task bundles a dataset, a prompt template, a scoring method (loglikelihood or generation), and a metric. Tasks are defined in YAML files, which makes adding new ones relatively lightweight. MMLU, for example, is split into 57 subtasks (one per academic subject), each with its own task definition.

Models are loaded via a backend abstraction. You can point lm-eval at an HF checkpoint, a local vLLM endpoint, an OpenAI-compatible API, or several other backends. The harness doesn't care how the model is served as long as it can call either loglikelihood(prompt, continuation) or generate_until(prompt, stop_tokens).

Results come back as a dictionary of metric values per task, which you can then aggregate or slice however you want.

The command-line interface looks something like this:

# 5-shot MMLU evaluation on Llama-3-70B via a local vLLM endpoint
lm_eval \
  --model local-completions \
  --model_args base_url=http://localhost:8000/v1,model=meta-llama/Llama-3-70B-Instruct \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size 16 \
  --output_path ./results/llama3-70b-mmlu \
  --log_samples
 
# The --log_samples flag saves each individual prompt+response,
# so you can audit exactly what the model saw. Highly recommended.

A few things about that invocation:

--num_fewshot 5 sets the shot count globally, but individual tasks can override it. MMLU's default in lm-eval is 5-shot; setting it to 0 will get you a genuinely different number.

--batch_size 16 affects throughput but not scores. Loglikelihood tasks are batched across all examples; generation tasks are batched up to the model's batch limit. If scores change with batch size, something is wrong.

--log_samples is the thing you should always include when running evals for anything you'll publish. Without it, you have a number but no way to debug it or reproduce it. With it, you can open a JSON file and see exactly what prompt the model received for any question, what it output, and whether it got credit. That's the audit trail that turns a number into a claim.

The prompt template problem

This is the part that isn't obvious until you see it.

MMLU is one of the most widely cited language model benchmarks: 57 academic subjects, 14,079 multiple-choice questions. But "MMLU score" isn't really one number. It's a family of numbers parameterized by harness choices. Same model, same 14,079 questions, different knobs, different percentages.

The three main knobs:

Shot count. 5-shot is conventional but not universal. 0-shot drops scores significantly on most models (anywhere from 2 to 15 points depending on the model). Some papers report 1-shot or 3-shot. The original MMLU paper used 5-shot, but plenty of evaluations ignore this.

Prompt template. The "raw" template just pastes the question. The "Q: ... Answer: A/B/C/D" format adds structural framing. Chat models often perform better with their native chat template applied. The performance difference can be more than 5 points.

Scoring method. As we covered above, loglikelihood vs. generate-and-extract can disagree by several percentage points on the same model.

Normalization. When doing loglikelihood scoring, do you normalize by the length of the option text? "Paris" is 5 characters; "The temperature decreases but the volume remains constant" is 60. Without normalization, longer options are systematically penalized because their log-probability is a sum over more tokens. With normalization, short options get a different kind of boost. Different harnesses normalize differently.

Each of these knobs is a coherent methodological choice with a defensible rationale. The problem isn't that any of them is wrong. The problem is that they're rarely documented clearly enough to let you know which combination was used when you read a headline.

HELM: holistic evaluation

HELM (Holistic Evaluation of Language Models) from Stanford took a different approach to the same problem. Instead of trying to standardize a single methodology, HELM explicitly frames evaluation as a matrix:

• Scenarios: the tasks (question answering, summarization, code completion, disinformation robustness, etc.)
• Metrics: accuracy, but also efficiency (tokens per dollar), robustness (stability across paraphrase), calibration (does the model's confidence match its accuracy?), fairness (does performance vary across demographic groups?), and harm potential
• Models: the models being compared, tracked over time

The "holistic" framing is what makes HELM different. In this picture, a model isn't summarized by a single accuracy percentage on MMLU. It's a point in a multidimensional space, and depending on your use case, different dimensions matter. A model deployed for consumer-facing medical information has very different requirements than one deployed for code completion.

In practice, HELM's breadth makes it expensive to run (you need to evaluate many scenarios × many metrics × many models), which is why it's mostly used as a periodic comprehensive audit rather than an everyday comparison tool. EleutherAI's harness is faster and more targeted.

HELM v2 (which is what's actively maintained as of 2026) distinguishes between "Classic" (the original set of scenarios from the 2022 paper) and "Lite" (a curated subset runnable in a few GPU-hours). If you see HELM scores cited, check which suite was used.

The replication problem

So now look at what happens when these knobs collide with the way models actually get evaluated in the wild. "GPT-4 gets 86% on MMLU" is a claim, and once you know the harness has a dozen freely chosen settings, that claim stops being a fact and starts being a measurement conditional on a configuration.

When OpenAI published GPT-4's technical report, they reported MMLU numbers. When academic groups tried to replicate those numbers using lm-eval-harness on GPT-4 via the API, they got different numbers. Some were lower, some unexpectedly different, and the differences were hard to fully account for without knowing exactly which prompt templates, shot counts, and normalization methods OpenAI used internally.

Every team builds its own internal eval pipeline, optimizes it for their models, and reports numbers from that pipeline. When someone else runs the same benchmark with different tooling, they get different numbers. Neither number is "wrong." They're measuring slightly different things.

The spread shown above isn't unusual. Alzahrani et al. showed that evaluation configuration choices flip leaderboard rankings: the "best" model on MMLU depends on which harness variant you believe, and the MMLU numbers they found vary by a comparable spread.

This matters a lot for comparing models to each other. If Model A was evaluated with configuration X and Model B with configuration Y, what you're really comparing is two configurations that happen to have different models attached. The model effect and the config effect are tangled together, and you can't untangle them unless both sides ran the same pipeline.

Under the hood: how tasks are defined

A task in lm-eval-harness is a YAML file. Here's a simplified version of what MMLU's task definition looks like:

A few things to notice.

output_type: multiple_choice is what enables loglikelihood scoring. Changing this to generate_until with a stop sequence and a regex metric would give you generation-based scoring, and a different score.

doc_to_text is the Jinja2 template that formats the question. Notice "Answer:" at the end with no space. That's a real choice: including the space, omitting it, or ending with just the option list affects which continuation the model assigns highest probability to.

fewshot_config: sampler: first_n takes the first N examples. Using random sampling from the dataset would give different examples (and different scores, especially for smaller few-shot counts).

All of this is visible and auditable if you're using lm-eval-harness and have --log_samples enabled. The problem is that many published numbers weren't produced this way. They came from internal eval pipelines whose configs weren't released.

Misconceptions

"A harness is just infrastructure; the real thing is the benchmark." The benchmark (MMLU, GSM8k, HellaSwag) is the questions and answers. The harness is what operationalizes those questions into actual model inputs and scoring logic. They're both doing real work. A benchmark with no harness is an untested hypothesis. A harness with no transparent methodology is a number without meaning.

"0-shot is a purer test because there's no information leakage from the few-shot examples." The few-shot examples don't reveal the right answer to the test question. What they do is establish the format the model should use for its response. Zero-shot often measures format sensitivity as much as it measures knowledge, which is why 5-shot is generally considered a more reliable probe of actual capability.

What's next

The next post covers contamination and leakage, how benchmark answers end up in training data, how to detect it, and why this problem is getting harder to solve as web-scale pretraining corpora and benchmark datasets increasingly overlap.

Additional reading (and watching)

• Gao, L., et al. (2021). A framework for few-shot language model evaluation. GitHub, EleutherAI. The primary reference for lm-eval-harness, including the loglikelihood vs. generation distinction.

• Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. The GPT-3 paper that introduced the few-shot evaluation setting most subsequent benchmarks inherited, including the 5-shot convention for MMLU.

• EleutherAI. (2024). lm-evaluation-harness. GitHub. Active codebase for the community-standard harness; the YAML task spec format is documented here.

• Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. The original MMLU paper: 57 academic subjects, 14,079 questions, and the 5-shot evaluation protocol that became conventional.

• Liang, P., et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110. HELM's defining paper, arguing for evaluation as a scenarios × metrics × models matrix rather than a single accuracy score.

• Bommasani, R., et al. (2023). HELM Lite: A Benchmark for Efficient Language Model Evaluation. Stanford CRFM. The streamlined subset of HELM designed for practical evaluation runs.

• OpenAI. (2023). GPT-4 Technical Report. arXiv:2303.08774. Contains the MMLU numbers used as the reference point for many subsequent comparisons, without full disclosure of the evaluation configuration.

• Alzahrani, N., et al. (2024). When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards. arXiv:2402.01781. Systematic study of how benchmark rankings change with evaluation configuration, illustrating the problem this post describes with empirical rigor.

• Open LLM Leaderboard v2. (2024). Hugging Face. The relaunched leaderboard with harder-to-contaminate benchmarks and full config transparency, motivated directly by the failures of the v1 leaderboard.