Code-generating LLMs reproduce buggy lines more often than fixes, and having seen buggy variants in training amplifies this bias; likelihood scoring usually favors fixes but some metrics flip preference when the bug alone was seen, suggesting memorised errors can skew evaluations and risk propagation in practice.

Main Finding

The paper introduces an exposure-aware evaluation framework and shows that prior exposure to buggy or fixed code in training data materially skews LLM behaviour and evaluation. While likelihood-based metrics (notably min/max token probability) tend to prefer the correct fixed code, LLM generations reproduce buggy lines far more often than fixes — especially when the model was exposed to the buggy variant during training. Exposure therefore can drive propagation of memorised errors and can confound evaluations of model correctness.

Key Points

• Exposure-aware pipeline: The authors sample single-line bug–fix pairs (ManySStuBs4J) and use membership testing (Data Portraits on Stack-v2) to determine whether the buggy and/or fixed variant was present in training data. Samples are stratified into four exposure classes: neither seen, both seen, only bug seen, only fix seen.
• Datasets: ManySStuBs4J (16,899 Java single-statement bug–fix pairs) and Stack-v2 (training corpus represented via a Data Portrait).
• Models evaluated: open models trained (at least partly) on Stack-v2 — StarCoder2 family (3B, 7B, 15B), Mellum-4B (open weights, partially undisclosed data but includes Stack-v2), and SmolLM3-3B.
• Membership testing (Data Portraits): uses strided Bloom filters with n-gram width w = 50 and stride s = 50; to avoid false negatives queries are padded to ≥99 tokens; badness score = fraction of n-grams found; threshold ≥90% badness used to mark a snippet as “seen”.
• Two evaluation modes:
  • Likelihood-based scoring from token conditional probabilities (various metrics: perplexity, mean token prob, min/max token prob, Gini coefficient, etc.).
  • Actual generation/completion behaviour and verbatim matching to bug or fix.
• Main empirical findings:
  • About 67% of SStuBs pairs have neither variant present in Stack-v2 (i.e., likely unseen).
  • When only one variant is present, fixes are more often present than bugs.
  • In generations, models reproduce buggy lines far more often than fixes. Exposure to the buggy variant markedly increases bug-regurgitation rates; exposure to fixes yields only marginal improvement in fix generation.
  • Likelihood metrics are heterogeneous: minimum and maximum token-probability metrics consistently prefer the fixed code across exposure conditions (a stable bias toward correct fixes), but distributional metrics like the Gini coefficient can reverse preference when only the buggy variant was seen.
  • Preferences vary by bug category: some categories are robust to exposure, others highly susceptible to memorised propagation; a few show intrinsic bug preference.
• Contributions:
  • An exposure-aware evaluation methodology for bug-vs-fix preference.
  • Empirical quantification of how exposure affects generation and likelihood-based scoring.
  • Analysis of metric sensitivity and breakdown by bug type.

Data & Methods

• Data:
  • ManySStuBs4J: 16,899 single-statement Java bug–fix pairs (processed with extra metadata).
  • Stack-v2: 68TB public code corpus; a 90GB Data Portrait (Bloom-filter-based) represents sampled 50-token n-grams.
• Membership inference:
  • Data Portraits check presence via hashed 50-token n-grams at 50-token stride.
  • To ensure sound queries, each SStuB snippet was padded to ≥99 tokens using surrounding context (padding on both sides to improve hit coverage).
  • Badness score is fraction of sampled n-grams matched; threshold of ≥90% marks a snippet as seen.
  • Caveats: Bloom filters can yield false positives (though tuned to ~0.1% FPR); stride/window construction can produce false negatives for short/unaligned snippets — mitigated by padding but still a methodological limitation.
• Evaluation pipeline:
  • Stratify SStuBs into four exposure groups: neither, both, only bug, only fix.
  • For each model and SStuB pair:
    • Compute conditional token probabilities for the bug and fix given identical preceding context.
    • Compute multiple metrics from the token-prob sequences (perplexity, mean token prob, min/max token prob, Gini coefficient, length baselines).
    • Generate completions under decoding; check verbatim match rates to bug or fix lines.
  • Aggregate comparisons across exposure strata and by bug category to study effects and robustness.
• Models: open-weight models up to 20B parameters; training-data openness varies (StarCoder2 fully open-data on Stack-v2; Mellum includes Stack-v2 but has closed components; SmolLM3 uses Stack-v2 among public sources).

Implications for AI Economics

• Measurement and Benchmarking:
  • Evaluation inflation risk: benchmark scores (and perceived competence) can be inflated or distorted when models memorise test items — especially when fixes are present in training data more often than bugs or vice versa. This undermines the reliability of benchmarks used for product differentiation, contracting, or procurement.
  • Metric choice matters: different likelihood metrics give divergent conclusions; purchasers and regulators should standardise metrics that are robust to memorisation effects (e.g., the paper finds min/max token-prob metrics more stable).
• Product Quality, Liability, and Externalities:
  • Propagation of memorised bugs creates negative externalities: widespread use of LLM-generated code that regurgitates buggy patterns can introduce systemic costs (security incidents, debugging time, downstream failures) that accrue across firms and projects.
  • Liability and risk pricing: service providers and customers need to price in the risk of memorised bugs. If models are more likely to reproduce observed buggy patterns, downstream debugging costs and potential legal exposure rise — affecting insurance, SLAs, and commercial terms.
• Incentives for Data Governance:
  • Providers have incentives to curate training data to reduce memorised harmful patterns, but this is costly. The study strengthens the economic argument for dataset hygiene, provenance tracking, and selective exclusion (or targeted mitigation) of known-bad code.
  • Data disclosure and auditing: economic actors (enterprises, auditors, regulators) may demand greater transparency on training corpora or access to exposure tests (e.g., Data Portraits) to assess memorisation risk. This creates market value for third-party auditing services and tools.
• Labor and Productivity Effects:
  • Short-term productivity gains from LLM-assisted coding could be offset by increased debugging and security-fix costs if models memorise and propagate bugs. Firms should consider investment in verification tooling and human oversight, affecting labor allocation between coding and QA.
• Market Design & Competition:
  • Open-data models permit reproducible exposure analyses (as in this paper); closed-data models make it harder to detect memorisation and thus may command different risk premia. This could influence procurement choices and market competition.
• Policy implications:
  • Regulating training-set hygiene or requiring disclosure of known exposures for sensitive domains (e.g., security-critical codebases) could reduce negative externalities but will impose compliance costs.
  • Standardising exposure-aware evaluation in model certification regimes can improve social welfare by aligning incentives for safer model deployment.

Limitations (relevant to economic interpretation) - Membership inference using Data Portraits is probabilistic (Bloom-filter FPs, stride-induced FNs mitigated by padding) — exposure labels are estimates, not absolute facts. - Analysis is limited to single-statement Java SStuBs and models up to 20B parameters; broader languages, multi-line bugs, and larger closed models may exhibit different dynamics. - Only specific metrics and matching criteria were tested; other evaluation designs could yield additional nuance.

Overall, the paper shows that training-data exposure is a first-order factor affecting whether LLMs “see” bugs and then “do” (regurgitate) them. For AI economics, that implies measurable downstream costs, altered incentives for data curation and disclosure, and the need for exposure-aware auditing and benchmarks when valuing, procuring, or regulating code-generating LLMs.