Leaderboard accuracy for frontier LLMs is materially inflated: about 13.8% of MMLU items show evidence of training-data exposure, boosting reported scores (particularly in STEM/professional subjects) and creating memorization fingerprints that can mislead buyers, investors and benchmark-driven procurement.

Main Finding

Leaderboards overstate frontier LLM “intelligence” because a large and structurally non‑uniform fraction of benchmark items have been seen (exactly or in paraphrase) during model training. Using three complementary, reproducible methods on six frontier models (GPT-4o, GPT-4o‑mini, DeepSeek‑R1, DeepSeek‑V3, Llama‑3.3‑70B, Qwen3‑235B) and MMLU items, the authors show (1) lexical/web contamination is widespread (Experiment 1), (2) model accuracy drops when surface wording is changed (Experiment 2), and (3) behavioral probes reveal internal memorization signals in most items (Experiment 3). The methods converge on the same subject ranking (STEM most contaminated), and the aggregate evidence implies that much of reported benchmark performance reflects memorization/pattern completion rather than robust generalization.

Key Points

• Experiment 1 (lexical/web scan): 513 MMLU questions sampled (9 per 57 subjects). Using an 8‑gram overlap ≥30% AND the correct answer present in web snippets, overall contamination = 13.8%. STEM highest (18.1%); some subjects much higher (Philosophy 66.7%, Anatomy 55.6%). Estimated Performance Gain (EPG) from contamination ≈ +0.040 overall (per ConTAM), up to +0.054 in STEM.
• Experiment 2 (paraphrase / indirect‑reference diagnostic): 100 MMLU items across six subjects; for each item authors generated paraphrased and indirect‑reference variants and tested all six models at temperature 0. Average accuracy drop when moving to indirect references = 7.0 percentage points; Law and Ethics each drop ≈ 19.8 pp (largest declines), matching high contamination in Experiment 1. Some models (e.g., GPT‑4o) are robust to surface change; others (Qwen3, Llama) show big drops.
• Experiment 3 (TS‑Guessing behavioral probe): TS‑Guessing applied to all 513 sampled MMLU items × 6 models with two tasks:
  • Option Mask (OM): ask model to reconstruct a masked wrong choice; primary metric is partial match (≥50% token overlap).
  • Word Mask (WM): blank a content word; measure exact reconstruction.
  • Flagging rule: OM partial ≥ 0.50 or WM exact = 1 → question flagged. Results: average combined flagging = 72.5% (far above random baselines). Average OM partial ≈ 45.8%; OM exact ≈ 14.2%; WM exact ≈ 47.4%. STEM again ranks highest under behavioral probe (≈55.9%).
• DeepSeek‑R1 anomaly: very low exact recall but very high OM partial (76.6%) and zero WM exact → “distributed memorization” signature (semantic/conceptual storage without verbatim surface retention). This explains its low original accuracy combined with little sensitivity to surface change.
• Multi‑method convergence: web detection, paraphrase sensitivity, and internal behavioral reconstruction produce consistent subject rankings and mutually reinforcing evidence that contamination is pervasive and heterogeneous across domains.
• Practical interpretation: reported high benchmark scores can be materially inflated by memorization/exposure; removing contaminated items reduces reported accuracy and narrows claimed gaps between models and humans.

Data & Methods

• Datasets and sampling
  • MMLU test split sampled: 513 questions (9 per 57 subjects) for Experiments 1 & 3; separate 100‑item sample (≈17 per subject across 6 subjects) for Experiment 2. Random seed 42; overlap between E1 and E2 avoided.
• Experiment 1 — Lexical contamination detection
  • Query method: first 150 characters of question issued to Tavily web search; top‑5 result snippets collected.
  • Contamination flag conditions: (a) fraction of question 8‑grams found in combined snippets > 0.30; AND (b) correct answer text present verbatim in snippets.
  • EPG computed: acc × contamination_rate × 0.4 per ConTAM (Singh et al. 2024).
  • Conservative lower‑bound estimate (web index not identical to pretraining corpora; excludes paraphrases).
• Experiment 2 — Paraphrase and indirect‑reference diagnostic
  • For each sampled question authors generated two variants using GPT‑4o:
    • Paraphrase: different wording, identical knowledge requirement and correct answer.
    • Indirect reference: key entities replaced by descriptions/associations.
  • All six models evaluated on original, paraphrase, indirect at temperature 0.
  • Main metric: accuracy change (original → paraphrase/indirect); subject‑level aggregation.
• Experiment 3 — TS‑Guessing behavioral probe
  • Two tasks for each question/model (6,156 probes per task):
    • Option Mask (OM): mask one wrong answer choice; model told which option is correct and asked to reconstruct masked wrong option. Report OM exact and OM partial (≥50% token overlap); partial is primary.
    • Word Mask (WM): blank a long content word in stem; model fills in missing word; report exact matches.
  • Flagging rule: OM partial ≥0.50 OR WM exact = 1 → question flagged as memorized.
  • Random baselines: OM exact ≈ 0%, OM partial ≈ 5% (empirical), WM exact ≈ ≈0% given vocabulary.
• Models tested: GPT‑4o, GPT‑4o‑mini, DeepSeek‑R1, DeepSeek‑V3, Llama‑3.3‑70B, Qwen3‑235B. All probes run via public APIs; procedures reproducible from open benchmarks and API calls.

Implications for AI Economics

• Measurement and valuation of capabilities
  • Benchmarks are a primary signal used by investors, customers, and markets to value model capabilities. Systematic contamination and memorization mean benchmark scores can materially overstate true generalization. Economic valuations, commercial contracts, and go‑to‑market claims that rely on unadjusted benchmark numbers risk mispricing and misallocation of capital.
  • Recommendation: discount headline benchmark performance by contamination‑adjusted EPGs or require reporting of decontaminated benchmarks and robustness measures when valuing models.
• Incentives and market structure
  • Leaderboard incentives encourage “benchmaxxing” (including evaluation items in training/fine‑tuning or overfitting to test distributions). This creates a moral‑hazard problem: providers can boost scores cheaply without delivering real capability improvements, distorting competition and reducing welfare from R&D spend.
  • Policy or procurement that rewards clean, demonstrable generalization (e.g., requiring contamination audits) would shift incentives toward genuine capability improvements.
• Deployment, liability, and sectoral risk
  • Sectors relying on high‑stakes performance (law, medicine, finance) are particularly affected: the paper shows Law and Ethics were most surface‑sensitive and contaminated. Claims that models “match experts” in these domains should be treated skeptically unless validated on contamination‑free or adversarially paraphrased testbeds. Misplaced trust can generate economic losses, liability exposure, and reputational damage.
• Competition and data transparency
  • Closed‑model releases limit the use of retrieval‑based contamination detection; behavioral probes (like TS‑Guessing) are necessary and can be standardized. Market participants and regulators should require or incentivize disclosure of training data provenance, contamination audits, and independent robustness testing as part of product certification.
• Labor markets and substitution claims
  • If benchmark outperformance is driven by memorization/exposure rather than transferable reasoning, claims about substitutability of human experts are overstated. Economic models forecasting labor displacement or productivity gains should incorporate a contamination‑adjusted estimate of real generalization capacity and the costs of building and validating contamination‑free evaluation.
• Regulatory and procurement implications
  • Governments and large buyers should require contamination audits (combining web‑index scans, paraphrase diagnostics, and behavioral probes), mandate reporting of EPG or similar impact metrics, and favor models with demonstrated generalization on decontaminated benchmarks.
• Practical steps for economic actors
  • Investors, procurement officers, and corporate buyers: demand contamination‑adjusted metrics, require adversarial/paraphrase robustness tests, and treat high leaderboard scores as preliminary signals, not sole evidence of capability.
  • Model developers: publish contamination audits, maintain decontaminated benchmarks (or variants), and invest in robustness testing to build credible claims—this can become a competitive differentiator.
  • Regulators: standardize contamination detection reporting, require independent audits for regulated applications, and include benchmark‑cleanliness as a factor in certification.

Short takeaway: public benchmark wins are necessary but not sufficient evidence of economically valuable AI capability. Contamination is widespread, uneven, and can materially inflate perceived competence—economic stakeholders should factor contamination adjustments and robustness testing into valuation, procurement, and policy.