Skip to content
Arc 8: Evaluation & Scientific Discipline

Contamination and Leakage

14 min read

Benchmark contamination is the situation where the questions a model is tested on have already appeared, verbatim or lightly paraphrased, in the pretraining corpus. When that happens, the accuracy number is measuring memorization of the test set rather than the capability the benchmark was supposed to probe. The score still looks like generalization.

If a model scores 90% on GSM8K but has likely seen those problems during training, what has anyone actually measured? I don't have a clean answer to that, which is most of why I wanted to work through this post.

Contamination is usually structural rather than malicious. Here's the shape of it.

Why test sets end up in pretraining data

The basic intuition: the internet is the training corpus. Benchmarks are on the internet.

When a research team publishes a new benchmark, they release a paper, post the data to a GitHub repo or Hugging Face dataset, and sometimes write a blog post walking through example questions. Within days, those questions are indexed by search engines. Within weeks, they appear in discussion threads, Stack Overflow answers, tutorials, and blog posts. Within months, they've been copied into dozens of secondary locations. And then the next training corpus crawl happens.

Common Crawl, the most widely used pretraining data source, crawls the web on a schedule. It doesn't know what's a benchmark question and what's a recipe. It sees text, and it collects it. Standard preprocessing pipelines apply deduplication, quality filters, and toxicity screens, but those filters were designed to remove near-duplicate documents and low-quality content, not to detect evaluation data.

So the questions go up, propagate, get crawled, land in a training mixture, and the model quietly memorizes the answers. At evaluation time it recalls instead of reasoning, and nothing in the pipeline flags that anything happened. A benchmark released in early 2024 could plausibly appear in a model trained in late 2024.

The leakage chain in practice

There are a few distinct mechanisms behind contamination, and they require different defenses.

Direct text leakage. The simplest case: the test question, exactly as written, appears in a training document. The model has (probably) memorized the answer. This is what gets detected by n-gram overlap methods, which we'll cover shortly.

Near-duplicate leakage. The question is paraphrased or slightly reformatted: different wording, same logical structure. Standard n-gram overlap misses this. The question-answer pair was functionally seen during training even though the exact token sequence wasn't.

Answer leakage without question leakage. A training document explains "the answer to question 42 on the GSM8K benchmark is 17" without including the question itself. The model can pick up a spurious association between a question's shape and its answer, which is leakage that helps on the benchmark and nowhere else.

Synthetic amplification. This one is newer and harder to detect. When models are used to generate synthetic training data, they can amplify any leaked examples they've already seen. If a contaminated GPT-4 generates 10,000 synthetic math problems "in the style of GSM8K," those synthetic problems will statistically cluster around the actual GSM8K distribution. The student model that trains on them inherits the contamination even if it never saw the original benchmark directly.

Detection methods

So contamination happens. How do we detect it? Four main approaches, each with different trade-offs.

N-gram overlap

The oldest and most straightforward method: slide a window of N consecutive tokens (typically 8–13 tokens) across both the benchmark question and the training corpus, and flag any exact match.

5-gram overlap scan
scanning corpus position 0 / 20
benchmark questioncorpus (scrolling under window)Janetsells3ducksfor$2eachandbuys4chickensfor$1each.Thepriceofeggsroseby10%inJanetsells3ducksfor$2eachmakingfarmersprofitabledespitecostsrisingoveralldemandremainedstrong5-gram windowno matchslide a 5-token window across corpus; flag any exact match as contaminated

A 5-gram window slides across the corpus. When it exactly matches a window from the benchmark question, both sides light up. Real detection uses 8–13-gram windows to reduce false positives; 5-gram matches have a lot more coincidental overlap.

The choice of N is a trade-off. Shorter windows (N=5) produce more false positives because short token sequences appear by coincidence. Longer windows (N=13) miss paraphrased versions of questions. The N=13 threshold Eleuther uses in lm-evaluation-harness is a pragmatic compromise, and as far as I can tell nobody defends it as principled.

The limitation: this only catches exact or near-exact matches. A rephrased question, a translated question, or a question that appears in a format different from the original benchmark text will slip through.

Canary strings

A proactive detection method: before releasing a benchmark, embed deliberately unusual strings (canary strings) into the test data. These are sequences like BENCHMARK-CANARY-XK7F3-CONTAMINATION-CHECK that have no reason to appear naturally anywhere on the web. If you later find that string in a model's training data, or get the model to regurgitate it from a prompt, you know the benchmark was included.

canary string lifecycle
① author embeds canary in test set
benchmark papertest.jsonl (private)BENCHMARK-CANARY-XK7F31 canary, 1 sourcethe open webarXiv · GitHub · blogs · mirrorscanary ×1Common Crawlingests 0/14 mirrorsmodel weightscanary baked inprobe: "complete → BENCHMARK-CANARY-"model: ▍the canary had no reason to exist anywhere — if the model can produce it, it saw the benchmark.

The canary starts as one line in the authors' private test.jsonl, spreads across mirrors and tutorials, gets swept into the next Common Crawl, ends up in a training mixture, and finally resurfaces when the trained model is prompted to continue the prefix. No step is surprising on its own; the surprise is how fast the chain runs end-to-end.

Canary strings are useful for proving contamination definitively, but they require coordination before the benchmark is released, and they don't help detect contamination in benchmarks that didn't use them.

Membership inference via log-probability

This is the approach I find most convincing. The core observation is that if a model has seen a sequence during training, it will typically assign higher probability to that sequence than to a similar sequence it hasn't seen.

Formally, given a test sequence ss with nn tokens, the per-token log-probability is:

LPT(s)=1n∑i=1nlog⁡pθ(si∣s1:i−1)\text{LPT}(s) = \frac{1}{n} \sum_{i=1}^{n} \log p_\theta(s_i \mid s_{1:i-1})

Equivalently, perplexity is PPL(s)=exp⁡(−LPT(s))\text{PPL}(s) = \exp(-\text{LPT}(s)). When a model has memorized a sequence, it assigns high probability to every token, so LPT\text{LPT} is high (close to 0) and PPL\text{PPL} is low (close to 1). Clean sentences the model hasn't memorized will have higher perplexity, reflecting the inherent uncertainty in predicting arbitrary text.

membership inference via perplexity
two populations by perplexity
countperplexity (per-token log-prob, lower = model more confident)151020304560clean (never seen)contaminated (memorized)model assigns highconfidence (low PPL)membership inference: if PPL is suspiciously low, the model likely saw this text during training

Two populations of text sequences: clean (never seen verbatim in training) and contaminated (memorized from training data). Clean text clusters around typical perplexity values; contaminated text spikes down toward PPL ~2–5. A threshold on PPL separates the two populations, but any threshold will produce some false positives (clean text with low PPL) and false negatives (contaminated text that wasn't strongly memorized).

The challenge is setting the threshold. Clean text with short, common phrases will naturally have low perplexity. Contaminated text that appears only once in training might not be strongly memorized. In practice, you compare against a calibration distribution: a held-out set of items from the same distribution as the test set but known not to appear in training. If the test items have suspiciously lower perplexity than the calibration items, that's evidence of contamination.

Oren et al.'s black-box test-order method

If a model has memorized a test set, it has memorized the items in whatever order they appeared in the training data, which is typically the order from the original benchmark file. So you can ask: does the model "predict" the correct ordering better than chance? Concretely, you take N test items and ask the model, in a prompt, which one comes first in the original ordering. If it has seen the test set, it will be right more often than random. If it hasn't, it should be roughly at chance.

The appeal is that you only need API access, no training data and no log-probabilities. The trade-off is that it only detects contamination when the model absorbed the ordering, which requires seeing the test set as a coherent document rather than as scattered examples across many training documents.

Worked example

GSM8K is a benchmark of 8,500 grade-school math problems released by OpenAI in 2021. It was designed to test multi-step numerical reasoning, with problems like "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?" (Answer: 18.)

When this benchmark was released, it was genuinely hard for models. A fine-tuned GPT-3 (175B) with a learned verifier scored around 55%, and without the verifier the baseline was much lower. By late 2023, scores above 90% were being reported across a range of models.

How much of that improvement is real? I don't know for certain, and nobody else seems to either. Several papers have identified GSM8K examples appearing in Common Crawl snapshots that were used for training. The exact overlap percentages vary depending on the n-gram threshold, but the presence of at least some contamination isn't disputed. What's disputed is how much it inflates scores, and that matters because GSM8K became one of the go-to benchmarks for comparing models.

Similar analysis has been done on MMLU, where specific questions have been found verbatim in WebText and other pretraining corpora. In some cases the question appears in an online quiz platform that was directly scraped. TruthfulQA has also been found in pretraining data for several models, though the nature of that benchmark makes contamination effects somewhat different (the "correct" answers require careful calibration rather than rote recall).

reported vs decontaminated score
as reported by paper
0%25%50%75%100%GSM8Kgrade-school math92%MMLU57-subject multi-choice86%HumanEvalPython code84%TruthfulQAmisconception traps72%GPQA-Dexpert science (private)54%reporteddecontaminated (13-gram filter)reported (ghost)illustrative drops across popular benchmarks — GPQA-D barely moves because its test set is kept private

What contamination does to the numbers. For each benchmark, the solid bar is the score a paper typically reports and the ghost outline is the score after a 13-gram decontamination pass removes items found in training data. The gap is your error bar on the reported number. GPQA-D barely moves because its test set is private.

And when you zoom out to how long each benchmark stayed uncontaminated before anyone caught it, the timeline is telling:

benchmark contamination lag
release → detected in training data
2016201720182019202020212022202320242025SQuAD40moMMLU26moGSM8K25moHumanEval17moTruthfulQA25moMATH18moreleasedcontamination detectedlag window

Release date (blue) versus estimated contamination detection date (red) for six major benchmarks. The red bar is the lag window between release and confirmed presence in pretraining data. More recent benchmarks contaminate faster because crawl frequency has increased and the community uploads data to public repositories more quickly.

The arms race

The pattern is pretty predictable at this point. A new benchmark gets released, the community adopts it as the standard measure for some capability, models improve, scores saturate, someone discovers the benchmark is contaminated, and the community moves on to a new benchmark that then gets contaminated in turn.

The cycle keeps getting shorter. In 2017–2019, a benchmark might stay reliably uncontaminated for a few years, because pretraining corpora were smaller and crawled less often. By 2022–2024, the lag between release and contamination was often under a year. By 2025, any benchmark published in plain text is in a race against the next training run.

Three things drive that. Training corpora keep getting bigger and more web-inclusive, and the move from RedPajama (1.2T tokens) to DCLM (2.5T) to the Dolma-level mixtures frontier labs use means more of the internet is in training data, including the niche repositories and academic hosting sites where benchmarks land. Crawl frequency keeps climbing: Common Crawl ships a snapshot every month or two, frontier labs run their own crawls more often than that, and the gap between a benchmark appearing online and appearing in a training snapshot has gone from months to weeks. And Hugging Face Datasets made publishing a benchmark trivial, which is good for the field and also means the benchmark is crawlable the same afternoon. Some newer benchmarks ask people not to upload them, which is hard to enforce once a paper is out with examples in it.

Synthetic amplification compounds all of it, for the reasons above.

Several structural fixes have been proposed and partially implemented.

Private test sets. Keep the test questions secret. Don't publish them. GPQA Diamond (graduate-level expert science questions) has moved toward this model, with careful controls over who can access the test questions and under what conditions. The limitation is that private test sets are harder to reproduce and audit, and they require trusting the organization holding the data.

Dynamic benchmarks. Instead of a static test set, continuously generate new questions. Dynabench takes this approach: models are evaluated against questions humans specifically designed to defeat the current best models. SWE-bench Verified uses GitHub issues that were filed after a specific date, so they couldn't have been in training data for models trained before that date. The limitation is that dynamic benchmarks are expensive to maintain and sometimes change in character over time, making longitudinal comparisons harder.

Contamination-aware evaluation protocols. Rather than trying to prevent contamination, measure and correct for it. Eleuther's lm-evaluation-harness has built-in n-gram decontamination support; many papers now report both raw scores and decontaminated scores where possible. MMLU-Redux is a manually re-examined version of MMLU that has corrected labeling errors and documented known contamination patterns.

None of these fully solve the problem. Private test sets are robust but opaque. Dynamic benchmarks are fresh but hard to compare. Contamination correction is useful but requires knowing what's contaminated. If I had to pick one to bet on, it'd be the dynamic benchmarks, mostly because they're the only option that doesn't require trusting somebody's word.

Implementation sketch: n-gram decontamination

Here's a minimal implementation of n-gram decontamination. This is conceptually what lm_eval does when you pass --decontamination_ngrams_path:

# --- Build n-gram index from training corpus ---
 
def build_ngram_index(corpus_path: str, n: int = 13) -> set[tuple]:
    """
    Read training corpus, tokenize naively by whitespace, and build
    a set of all n-grams. This is the offline step; run once and cache.
    """
    ngrams = set()
    with open(corpus_path) as f:
        for line in f:
            tokens = line.strip().split()
            for i in range(len(tokens) - n + 1):
                gram = tuple(tokens[i : i + n])
                ngrams.add(gram)
    return ngrams
 
# --- Check benchmark questions for contamination ---
 
def is_contaminated(
    question: str,
    ngram_index: set[tuple],
    n: int = 13,
    threshold: int = 1,  # flag if >= this many matching n-grams
) -> bool:
    """
    Returns True if the question shares >= threshold n-grams with the
    training corpus. Lower threshold = more sensitive = more false positives.
    """
    tokens = question.split()
    if len(tokens) < n:
        return False  # too short to form a valid n-gram
 
    matches = 0
    for i in range(len(tokens) - n + 1):
        gram = tuple(tokens[i : i + n])
        if gram in ngram_index:
            matches += 1
        if matches >= threshold:
            return True
    return False
 
# --- Example usage ---
 
# index = build_ngram_index("pile_train.txt", n=13)
# question = "Janet's ducks lay 16 eggs per day. She eats three for breakfast..."
# print(is_contaminated(question, index))  # True if verbatim overlap found

A few things to notice. The n-gram method is completely agnostic to what the training corpus contains; you just need a file with the text, which is why it's the one that actually gets run. Membership inference needs model access and a calibration set, a much higher bar. Both have meaningful false positive and false negative rates, so I'd run both and flag any item that either one catches.

Misconceptions

"High scores on contaminated benchmarks mean nothing." This overstates it in the other direction. Even if a model has seen some GSM8K questions, getting 90% correct still requires some underlying mathematical reasoning capability. You can't memorize your way to that score if the contamination rate is, say, 5–10% of questions. My read: contamination inflates scores, probably meaningfully for heavily contaminated benchmarks, but doesn't completely explain high performance on tests with complex reasoning requirements.

"The solution is just to never publish test sets online." Unpublished test sets solve the direct leakage problem and create new ones: they're harder to audit, harder to reproduce, and they give the organization holding the test set room to structure evaluations favorably. Transparency about methodology is worth something even though it carries contamination risk. The better fix is some combination of private tests, dynamic evaluation, and honest reporting with contamination estimates.

"Synthetic data doesn't create contamination problems." This was arguably true in 2022, when synthetic data was rare. It isn't true now, for the amplification reasons above, and there's no clean solution yet.

What's next

Contamination is ultimately a symptom of a deeper problem. Evaluating a model on data that looks like what it trained on doesn't tell you whether it has generalized, and better decontamination only helps at the margin. The harder question is what we're measuring and why.

The next post covers evaluating reasoning, which is where this lands in practice: why multiple-choice accuracy on MMLU tells you almost nothing about genuine reasoning capability, and what newer evals like GPQA and FrontierMath are attempting to measure instead.

Additional reading (and watching)