Human-friendly code survives AI edits better: a study of 5,000 Python competitive-programming files finds higher CodeHealth predicts stronger semantic preservation after LLM refactoring, implying maintainability lowers AI intervention risk and can guide where human oversight is most needed.

Main Finding

Human-oriented maintainability (measured by CodeHealth, CH) predicts how safely LLMs can refactor code: files in the "Healthy" CH range (CH ≥ 9) have materially lower AI-refactoring break rates (15–30% risk reduction). CodeHealth outperforms intrinsic LLM confidence (perplexity, PPL) and simple size (SLOC) as a predictor of whether an LLM refactoring preserves behavior.

Key Points

• Primary result: Higher CodeHealth → lower probability that an LLM refactoring breaks tests. Healthy files (CH ≥ 9) show statistically significant lower break rates.
• CH is a better predictor of refactoring correctness than:
  • Perplexity (PPL): PPL differences between Healthy and Unhealthy code are inconsistent across models and have negligible effect sizes; PPL is largely orthogonal to CH.
  • Source Lines of Code (SLOC): CH adds predictive power beyond file size.
• Quantitative risk reduction reported: roughly 15–30% lower risk of breaking tests on Healthy vs Unhealthy files (depending on model).
• Studied LLMs: five medium-sized open models (20–32B parameters) run locally and one state-of-the-art agentic model (Anthropic Claude Sonnet). Sampling temperature set to 0.7; generation capped at 8,192 tokens.
• Refactoring task design: LLMs were asked to refactor Python files for maintainability while preserving behavior; unit tests accompanying competitive programming solutions served as the oracle for semantic preservation.
• Common code smells in the dataset: nested control complexity, complex methods, deep nested complexity, complex conditionals, excessive function arguments.
• Predictive modeling: shallow decision trees (max depth 3) and logistic regression used for interpretability; CH was a primary decision node and boosted AUC relative to alternatives.

Data & Methods

• Dataset:
  • Source: Google DeepMind CodeContests.
  • Sample: 5,000 Python solutions (2,500 labeled Healthy, 2,500 Unhealthy), filtered for 60–120 SLOC and at least one CodeScene-detected smell; ensured diversity via CodeBleu similarity filtering.
  • For cost-controlled experiments with Claude Sonnet, a 1,000-sample subset was used (500 Healthy, 500 Unhealthy).
• CodeHealth (CH):
  • File-level score 1–10 computed by CodeScene from 25 Python code smells. Categories: Healthy (CH ≥ 9), Warning (4 ≤ CH < 9), Alert (CH < 4).
• RQ1 (Perplexity):
  • PPL extracted for five Hugging Face models; outliers (hard-coded enormous literals) filtered via robust z-score; distributions non-normal → Mann–Whitney U tests with Holm correction; effect sizes reported with Cliff’s δ.
• RQ2 (Refactoring break rate):
  • Uniform refactoring prompt used for all LLMs; outputs run against original unit tests; a refactoring is considered broken if any test failed post-refactor.
  • Compared break rates between Healthy and Unhealthy using chi-square tests; reported Risk Difference (RD) and Relative Risk (RR) with 95% CIs; Holm correction for multiple tests.
• RQ3 (Predictive power):
  • Features: CodeHealth, Perplexity, SLOC (token count excluded to reduce redundancy).
  • Models: shallow decision tree (max depth 3, min leaf 25, class-weighted), 5-fold cross-validation; logistic regression as robustness check (reported Odds Ratios). Reported AUC as threshold-independent performance metric.
• Models/experimental settings:
  • Medium-sized LLMs: Gemma, GLM, Granite, GPT-OSS, Qwen (≈20–32B); Sonnet = Anthropic Claude Sonnet-4 (SotA agent) used on subset.
  • Sampling temperature = 0.7; generation cap = 8,192 tokens.

Implications for AI Economics

• Operational risk and cost management
  • Use CH to triage where to apply automated AI interventions vs. where human oversight is needed. Targeting AI on high-CH modules reduces break/fix costs and lowers operational risk when scaling LLM-based tools.
  • Quantifiable risk reduction (15–30%) can be incorporated into cost-benefit and risk models for rolling out AI assistants: healthier codebases yield better marginal returns on AI automation.
• Investment prioritization and ROI
  • Investments in maintainability (refactoring, reducing code smells) pay double dividends: they reduce human maintenance effort and materially improve compatibility with AI tooling, improving expected ROI of AI adoption.
  • CH can serve as an objective signal to prioritize refactoring spend where it unlocks safe automation.
• Governance, procurement, and tooling
  • CodeHealth can be adopted as a measurable governance metric to define service-level expectations for AI-assisted coding (e.g., which modules are “AI-eligible”).
  • Vendors and procurement teams can use CH thresholds to specify acceptable risk levels for automated changes and to negotiate support/monitoring terms.
• Workforce and productivity effects
  • Organizations with better-maintained code may be able to shift more routine refactorings to AI assistants, freeing developers for higher-value work. Conversely, poor-quality code constrains AI utility and may require continued manual effort, affecting productivity forecasts.
• Limits, caveats, and areas requiring caution
  • Dataset bias: study uses competitive programming solutions (well-tested, small, algorithmic code) — results may not fully generalize to large, multi-file, dependency-rich production systems.
  • Task scope: focused on refactoring (preserve behavior); other AI tasks (feature addition, bug fix, design changes) may behave differently.
  • Model & prompt dependence: results are conditional on prompt formulation, model versions, and hyperparameters (temperature, token caps); agentic behaviors and tool access (e.g., web or repo access) change outcomes.
  • Measurement and governance overhead: enactment requires tooling (CodeScene or equivalent) and integration into CI/CD and procurement processes — there is an upfront cost.
• Recommended organizational actions
  • Start measuring CH (or an equivalent maintainability metric) across codebase to identify low-risk targets for AI automation.
  • Pilot AI refactoring on Healthy modules first to validate safety and yield early wins.
  • Use CH thresholds to design tiered review policies: fully automated for high CH, human-in-the-loop for medium CH, manual-only for low CH.
  • Incorporate CH-based benefits into ROI analyses for AI tooling investments and training programs.

Limitations noted by the authors (relevant for economic assessment): Python-only, medium-sized single-file problems from competitive programming, limited agentic experimentation (cost-limited sample for Claude), and evolution of LLM capabilities over time — estimates should be updated as models and deployment contexts change.