Contamination and Leakage

A held-out test is meant to ask how a model handles examples it was not trained to answer. That boundary becomes difficult to maintain when the questions, solutions, and discussions are public and training corpora draw from the web.

The problem is called test-set contamination. It can happen during pretraining, instruction tuning, or another training stage. Repeatedly choosing models and prompts against the same test set can also overfit the evaluation, even without placing those items directly in the training batches.

The evaluation procedure may be reproducible even when test material appeared in training. Establishing that exposure and measuring its effect on a score require different evidence.

Following the material into training

A benchmark might live in a repository, appear in a dataset mirror, and acquire worked solutions in tutorials. A collection pipeline could ingest any of those copies. Filtering might remove some and miss others. Public availability establishes an opportunity for exposure; it does not prove inclusion in a particular model’s training run.

The relevant unit is not always the entire question. An answer key, distinctive solution, or paraphrase may reveal enough to make an item easier. On the other hand, encountering related subject matter is expected: a chemistry model should learn chemistry. Training on the same topic is not automatically leakage of the particular held-out example.

Synthetic data adds another route. A teacher can reproduce an existing test item or solution. Merely generating new problems of a similar kind does not demonstrate contamination. To make a specific claim, identify the shared content and the path by which it could have entered training.

Exact overlap is evidence with a scope

An n-gram detector looks for contiguous sequences of $n$ tokens or words shared between an evaluation item and a corpus. Long distinctive matches can be useful leads. Common phrases create false positives; punctuation changes, paraphrases, and translated copies can evade an exact matcher.

The normalization rule belongs in the report. Lowercasing, Unicode normalization, punctuation removal, and tokenization all change which strings match. A search over only part of the training corpus can only support conclusions about that part.

Here is a deliberately small detector. It reports whether exact normalized word n-grams overlap; it does not label unmatched text “clean.” Inputs shorter than the window receive a separate status.

For a large corpus, building an in-memory Python set like this is expensive. Use an indexed or streaming implementation appropriate to the corpus, preserve document boundaries, and review flagged spans.

Canaries and likelihood tests

A canary is a marker deliberately inserted where its later appearance can be traced. Its value comes from controlled provenance. If a marker is printed in the evaluation prompt, retrieved by a tool, or widely discussed elsewhere, reproducing it does not isolate training exposure.

Likelihood-based membership inference asks whether a model assigns unusually high probability to a candidate training example. That can provide evidence under a validated method, but easy unseen text can also have high probability, and a training member need not have uniformly high token probabilities. There is no general perplexity cutoff that separates members from nonmembers across models and domains.

Oren et al.’s test uses a more specific signal. Under an exchangeability assumption, an uncontaminated model should not prefer a benchmark’s arbitrary canonical ordering over shuffled orderings. Their procedure compares the likelihood of the canonical sequence of examples with likelihoods after permutation.

This does not require the model weights or training corpus, but it does require likelihood access and the method’s statistical assumptions. It is not a conversational test asking which question appeared first. Natural ordering structure could invalidate exchangeability, and failure to detect the signal does not establish absence of all contamination.

A filtered score is not a causal estimate

Suppose a model answers seven of ten questions correctly. An overlap rule flags two questions, both answered correctly. Removing them leaves five correct out of eight, or 62.5%, down from 70%.

To attribute the 7.5-point difference to contamination, we would need to know how the same model performs on comparable questions without the exposure. The flagged subset might simply be easier, more conventional, or from a different subject. Filtering changes the test population at the same time it changes exposure risk.

Report the rule, how many items it removes, and the scores on both subsets. Inspect item difficulty and subject composition. If possible, use controlled training experiments or independently constructed matched items to investigate score inflation.

Keeping an evaluation useful

Private holdouts reduce public exposure, but access needs to remain controlled and the items need to represent the intended task. Fresh public problems can help when their publication dates are after a model’s relevant training stages. Dates alone are weaker evidence if the model uses retrieval, its training cutoff is unknown, or an earlier copy existed elsewhere.

LiveCodeBench uses continuously collected contest problems and time-based evaluation to address this issue for code. Record the exact release window and model version rather than treating a benchmark name as a permanent guarantee of independence.

Other benchmark revisions solve different problems. SWE-bench Verified is a human-filtered subset intended to improve task quality and evaluability; “Verified” does not mean that every issue is newer than a model’s training data. GPQA is a difficult science benchmark, but its published questions are not a private holdout just because they are hard.

A useful contamination report distinguishes corpus overlap, evidence of model exposure, and demonstrated effects on performance. New independent test items from the intended distribution remain useful; generalization does not require changing the entire task distribution each time a benchmark becomes public.

The next post looks at another boundary in what a score can establish: evaluating reasoning.

Additional reading

• Oren et al., Proving Test Set Contamination in Black Box Language Models. Likelihood-based permutation testing and its exchangeability assumption.
• Jain et al., LiveCodeBench. Time-based evaluation with newly collected coding problems.
• SWE-bench. Benchmark variants, including the human-filtered Verified subset.
• Rein et al., GPQA. Construction of the expert-written multiple-choice dataset.