Superficial code rewrites slightly weaken AI coding agents: semantics-preserving perturbations cut repair success by up to 6.7 percentage points and raise effort, and model rankings by robustness change with the agent scaffold and repository.

Main Finding

Local, semantics-preserving code rewrites (semantics-preserving transformations, SPTs) cause measurable but heterogeneous drops in the effectiveness and efficiency of repository-level AI code agents. Across 54 repair instances and 16 (scaffold × model × benchmark) configurations, perturbations produced up to a 6.7 percentage-point mean drop in resolve rate in the worst configurations, with statistically significant degradation in 6 of 16 configurations. Perturbations also increased agent effort—step counts and token cost rose by up to ~9.9% and ~22.9% respectively—even when resolve rates were largely unchanged. Robustness is not a fixed property of a model: a model’s robustness ranking changes with scaffold and workload, producing a “jagged robustness frontier.”

Key Points

• Scope and targets
  • Evaluated repository-level issue resolution (FAIL_TO_PASS and PASS_TO_PASS test oracles) on 54 instances drawn from SWE-bench Verified (28 instances) and SWE-bench Pro (26 instances).
  • Two agentic scaffolds: mini-SWE agent and OpenCode.
  • Four frontier models: Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, Qwen 3.6-27B.
• Semantics-preserving transformations (SPTs)
  • Catalog of 14 SPTs: control-flow rewrites (if/else swap/negation, loop rewrites/unrolling, condition splitting), operand/ comparison swaps, double-negation injection, commutative reordering, local renaming, try/except wrapping, dead-code injection (unreachable blocks, dead method, unread string assignment), string literal splitting, etc.
  • Transformations validated by differential testing on large test suites for SymPy, sqlfluff, and xarray (no test outcomes changed in those validations).
  • Perturbations exclude test files and preserve the gold-patch files to keep tasks comparable.
• Sampling design
  • Random Variant Sampler (non-adversarial): for each seed repository, produce N = 20 variants by sampling Nt = 3 transformations per variant, selecting files with per-transformation random inclusion pfile ∼ U(0,1), applying transforms to ϕ = 0.7 fraction of matched sites, Nk = 5 keyword bindings, and cap Nf = 10 files per keyword-bound transform.
  • Keyword targets for some transforms are extracted from the issue description via a single LLM call; sampler is intentionally not feedback-guided (gives lower-bound effects).
  • Median perturbation magnitude: targets ≈ 6.9% of source lines (Verified) and 7.7% (Pro).
• Experimental protocol and statistics
  • For each instance: 20 unperturbed runs on the seed and 20 perturbed runs (one per variant), each in an isolated environment. This pairing isolates perturbation effects from agent stochasticity.
  • Primary metric: per-instance degradation Δ = r0 − rp (baseline minus perturbed resolve rates); configuration summaries use mean Δ across instances.
  • Effort metrics: relative changes in mean agent steps (δstep) and token cost (δcost).
  • Statistical inference: fixed-population approach, 95% bootstrap percentile intervals (B = 20,000) for configuration-level means; Newcombe intervals for per-instance proportions.
• Empirical findings
  • Resolve-rate impact: generally small but non-negligible; up to 6.7 percentage-point mean drop in the most affected configurations; 6 of 16 configurations showed statistically significant degradation.
  • Effort/cost impact: average increases in steps and token cost (up to ~9.9% more steps and ~22.9% higher token costs), so outcome-only metrics understate operational impact.
  • Model × scaffold × workload interaction: no stable robustness ordering of models across scaffolds. Example: Qwen was among the most robust under mini-SWE agent on SWE-bench Verified but the most brittle under OpenCode.
  • Simpler scaffolds (mini-SWE agent) tended to be more robust to SPTs than more complex scaffolds (OpenCode).
  • Heterogeneity across instances: perturbation effects concentrate on a subset of repositories (large instance-specific variance).
• Reproducibility: code and sampler implementation available at the project GitHub.

Data & Methods

• Benchmarks and instances
  • SWE-bench Verified (human-validated subset of SWE-bench) and SWE-bench Pro; total 54 chosen instances (28 Verified, 26 Pro) due to compute constraint.
  • Each instance includes a seed repo, an issue description, the gold patch, and test suites defining success.
• Agent/scaffold configurations
  • Two scaffolds: mini-SWE agent and OpenCode.
  • Four models: Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, Qwen 3.6-27B.
  • Crossed to give 16 independent configurations.
• Variant generation
  • Random Variant Sampler parameters: N=20 variants/instance, Nt=3 transforms/variant, Nk=5 keywords, Nf=10 files cap for keyword-bound transforms, ϕ=0.7 site fraction.
  • Transformations applied only to source files (tests excluded); transforms validated individually on large test suites for three representative projects.
  • Sampler is intentionally non-adversarial (no feedback loop from model results).
• Run protocol
  • For each instance and configuration: run agent 20 times on the seed; generate 20 variants and run agent once per variant; evaluate patches against the same test oracle.
  • Isolated environments per run; identical agent settings across seed and variants.
• Metrics and inference
  • Resolve rate, per-instance degradation Δ = r0 − rp, mean degradation across instances.
  • Effort: relative change in agent steps and token cost per instance, averaged.
  • Statistical treatment: fixed-population; bootstrap percentile intervals for configuration means; Newcombe intervals for differences of proportions; analysis separates agent stochasticity vs. variant variability.

Implications for AI Economics

• Benchmarking and procurement
  • Standard capability benchmarks can overstate "deployment reliability." Rankings based purely on unperturbed benchmark performance do not reliably predict real-world robustness; procurement decisions based solely on benchmark scores risk suboptimal choices.
  • Buyers should require robustness evaluations that vary scaffolds and codebase presentations, not just single-run benchmark numbers.
• Operational costs and pricing
  • Semantics-preserving variability raises expected operational costs: more agent steps and higher token usage increase per-issue compute cost and latency—even when success rates are similar.
  • Firms should incorporate an expected-cost premium for brittleness (mean extra tokens/rounds) and consider tail-risk pricing for rare but costly instance-specific failures.
  • SLA design should include robustness metrics (mean and tail behavior) and account for non-uniformity across workloads.
• Vendor selection and contracting
  • Because robustness depends jointly on model, scaffold, and workload, economic decisions must consider the whole pipeline (model + agent scaffold + expected codebase characteristics). Contracts should specify which scaffold + model combination was validated and for which workload types.
  • There is value in simple, predictable scaffolds (lower complexity can reduce brittleness), so investment in simpler scaffold designs can yield economic returns through reduced rework and lower token costs.
• Testing, risk assessment, and QA
  • Organizations should add non-adversarial SPT-driven stress tests (and adversarial variants for upper-bound risk) to pre-deployment QA to measure expected degradation and cost increases per workload.
  • Measure instance-level heterogeneity and tail risk: a small number of brittle instances can dominate operational losses—economic models should capture this concentration.
• Incentives for model and scaffold developers
  • Model makers should provide robustness evaluations across agent scaffolds and representative codebase perturbations; scaffold builders should design for invariance to typical refactorings and inert code.
  • There is an economic case for investing in robustness diagnostics, canonicalization layers (normalize code representations), and retrieval/localization methods that are resistant to decoys (dead-code injections, renames).
• Policy and standards
  • Robustness to semantics-preserving transformations can be formalized as a deployment requirement or labelling metric. Regulators or industry bodies could standardize tests that better reflect operational reliability, reducing asymmetric information in procurement.
• Practical short recommendations for stakeholders
  • For buyers: demand robustness tests that include SPT-like perturbations, evaluate end-to-end scaffold + model combos, budget for higher-than-benchmark operational costs, and require SLAs covering robustness metrics.
  • For vendors: publish robustness profiles across scaffolds and workloads, optimize scaffolds for simpler, more robust interaction patterns, and provide guidance on workload characteristics that exacerbate brittleness.
  • For economists/analysts: incorporate expected extra compute cost, failure concentration (tail risk), and reduced reliability into cost–benefit and ROI analyses for AI-assisted software engineering tools.

Code and artifacts: the authors provide the Random Variant Sampler and experiments on GitHub (repository linked in paper).