How Benchmarks Get Gamed

Why this matters for procurement

A vendor presents a benchmark result. The number looks impressive. Your team asks you whether to proceed. The honest answer is: you cannot evaluate that number without knowing how it was produced.

Benchmark gaming is not fraud in the legal sense. Most of it is technically defensible. It is a set of practices that produce numbers that look like evidence of capability while systematically overstating performance on real tasks. The five patterns below account for the majority of misleading claims in enterprise AI sales cycles.

Pattern 1: Data contamination

A model is trained on data that includes, in some form, the questions or tasks it will later be evaluated on. When this happens, the model is not demonstrating capability : it is demonstrating memorization.

Data contamination is especially common in large language models trained on internet-scale datasets. Benchmark questions, along with their answers, appear in blog posts, forums, and educational websites. A model trained on this data has almost certainly seen variations of the test questions before the test begins.

Detection signal: Ask the vendor: "What contamination analysis did you run against this benchmark's training split?" A credible answer describes a specific methodology. No answer, or "we use standard data cleaning," is a red flag. Research labs publishing rigorous papers (Guo et al., arXiv:2406.00515) have shown significant contamination in widely used benchmarks, with models achieving high scores partly through memorization rather than generalization.

Pattern 2: Task overfitting

A model is fine-tuned or prompted specifically on the format and style of a benchmark. It learns to be good at that benchmark's specific task structure, without generalizing to the broader capability the benchmark was designed to proxy.

The distinction matters: a model fine-tuned on math competition formatting might score higher on GSM8K not because it understands mathematics better, but because it learned the output format that evaluation scripts score as correct. When you deploy it on your actual business math problems : formatted differently, with different variable names and context : the score does not transfer.

Detection signal: Ask for performance on held-out tasks from the same capability domain that were not used in fine-tuning. If the vendor cannot provide this, the fine-tuning evaluation is closed-loop and untrustworthy.

Pattern 3: Cherry-picked splits

A benchmark has hundreds or thousands of tasks. A researcher evaluates multiple model versions across different subsets and reports results from the subset where the best model performed best. The subset was chosen after seeing the results.

This is a form of p-hacking applied to AI evaluation. The reported performance is technically real : the model did achieve that score on that subset. But the subset was not chosen in advance. It was identified after evaluation as the subset where the number looked best.

Detection signal: The results section should specify the evaluation split before showing numbers. If the paper or vendor report shows performance only on a named subset without explaining how that subset was defined prior to evaluation, treat the number as a selected best-case, not an overall result.

Pattern 4: Selective reporting

A model is evaluated on multiple benchmarks. The results that make the model look best are published. Results where the model underperforms are not mentioned.

Selective reporting is arguably the most common form of benchmark gaming because it requires no methodological irregularity. Every number reported is real. The problem is the numerators are chosen from a large pool of evaluations that included failures.

Detection signal: Compare the benchmarks reported in a vendor document against the standard benchmark suite for that capability category. If multiple standard benchmarks are absent without explanation, ask: "What other benchmarks were run that are not in this report?"

Pattern 5: Moving goalposts

A benchmark becomes saturated. Models achieve near-ceiling performance. Rather than acknowledging the ceiling, researchers introduce harder benchmark variants, reframe earlier results against the new harder standard, or claim the original benchmark was always insufficiently challenging.

This creates a comparison problem. A model that scored 90% on MMLU (now considered insufficient) cannot be directly compared to a model that scores 75% on MMLU-Pro (the harder successor). The numbers are on different scales measuring different things. Claims of "leading performance" made across benchmark generations are not directly comparable.

Huang et al. (arXiv:2406.04127) document the saturation dynamic explicitly: as benchmark completion rates approach ceiling, the discriminative value of the benchmark collapses, and comparative claims between models become unreliable.

Detection signal: When a vendor claims performance on a benchmark that has been publicly discussed as saturated, ask which version of the benchmark was used and what the current ceiling is. A result of 90% on a benchmark where top models score 92-95% is not a meaningful differentiator.

The detection checklist

What legitimate evaluation looks like

Rigorous AI evaluation pre-registers the benchmarks before training begins, uses held-out private evaluation sets alongside public benchmarks, reports all benchmarks run (not only those with strong results), and acknowledges limitations explicitly in the results section.

When a paper or vendor report does all of this, the benchmark results carry more weight. None of it guarantees transfer to your deployment context : but it removes the most common sources of artificial inflation.