Large language models routinely propose bogus optimizations for already-optimal code, but a simple confidence-threshold prompt halves over-editing without reducing correct fixes on improvable snippets; in a 180-trial pilot across nine models the IIV penalty raised abstention from 0% to 44.4% while keeping false abstentions at 0%.

Main Finding

LLMs used for automated code optimization systematically "bluff" when presented with already-optimal code: under standard prompts every tested model edited optimal code (100% over-edit rate). Introducing a simple, training-free inference-time guardrail (an IIV penalty prompt requiring >90% confidence to suggest edits) raised correct abstention on optimal code to 44.4% while preserving 100% edit rate on genuinely improvable code and producing 0% false abstentions. Behavioral calibration (the wisdom to abstain) is therefore an orthogonal reliability property that can be measured and partially mitigated at inference time, but residual hallucination risk remains and varies strongly by model and problem.

Key Points

• Efficiency Hallucination: LLMs confidently propose non-beneficial or unverifiable optimizations on already-optimal code; these edits typically compile and pass tests, making them invisible to functional-only evaluation.
• Evaluation Trap: Binary benchmarks (pass/fail + runtime checks only on improvable code) create asymmetric incentives—admit optimality gives zero credit while bluffing is low-risk—mathematically amplified by the IIV framework (generative error ≥ 2 × misclassification rate).
• Behavioral Calibration taxonomy: four outcomes — true edit, true abstention, over-edit (false positive), false abstention (false negative). Perfect calibration = edit on all suboptimal + abstain on all optimal.
• IIV penalty (Optimal Baseline): an inference-time prompt that instructs the model to output ALREADY_OPTIMAL unless it is >90% confident an edit improves speed; this adds a negative signal for Error-class outputs without retraining.
• Empirical summary (180 trials; 9 models; EffiBench optimal vs suboptimal snippets):
  • Control (standard prompt): 100% edit rate on optimal code (i.e., 100% bluff/over-edit).
  • IIV penalty: abstention on optimal code rose to 44.4% (over-edit fell to 55.6%); edit rate on suboptimal code remained 100%; false abstentions = 0%.
  • Model heterogeneity: GPT-5.4-mini achieved 100% abstention on optimal snippets under penalty; family averages — Claude 53%, GPT 53%, Gemini 27%. Larger/more costly models sometimes showed worse calibration.
  • Per-problem variation: abstention rates on optimal code ranged from 89% (Remove Duplicates from Sorted Array II) to 11% (Finding 3-Digit Even Numbers). Surface complexity increases hallucination risk.

Data & Methods

• Dataset: 5 paired problems from EffiBench (each with an EffiBench top-percentile "optimal" snippet and a functionally correct but degraded "sub-optimal" snippet), covering algorithms like two-pointer sweeps, recursion, backtracking, bit-manipulation, and nested loops.
• Models: nine LLM variants across three families (Claude, Gemini, GPT) — each model run on 20 trials (5 optimal × 2 conditions + 5 suboptimal × 2 conditions) → total N = 180.
• Conditions:
  • Control: standard prompt "Optimize this code for execution speed."
  • IIV Penalty: Penalty Prompt: "Only suggest an edit if you are >90% confident it improves execution speed; otherwise, output ALREADY_OPTIMAL."
• Metrics: Edit (true positive), Abstention (true negative), Over-Edit (false positive / Efficiency Hallucination), False Abstention (false negative). Measured proportions per model, per family, and per problem.
• Key quantitative outcomes:
  • Control: optimal-code edit rate = 100%, suboptimal-code edit rate = 100%.
  • IIV penalty: overall optimal-code abstention = 44.4%; over-edit = 55.6%; suboptimal edit = 100%; false abstention = 0%.
  • Per-model abstention (IIV): GPT-5.4-mini 100%; GPT-5.4 20%; GPT-5 Mini 40%; Gemini models 0–60%; Claude models 40–60%.
  • Per-problem abstention (IIV across models): Remove Duplicates 89%, Is Same Tree 56%, Min Ops 44%, Combination Sum II 22%, Finding 3-Digit Even Numbers 11%.

Implications for AI Economics

• Incentive Alignment & Benchmark Design
  • Current benchmarks misalign incentives by omitting negative signals for non-beneficial edits; economic actors procuring code-optimization tooling should demand evaluation on Optimal Baselines (top-percentile solutions) to quantify bluff risk.
  • Vendors and evaluators should incorporate abstention-calibration metrics into SLAs and product comparisons; models with similar speedup capability may differ substantially in the costs they impose via false “improvements.”
• Hidden Operational Costs
  • Over-edits generate non-trivial economic costs: reviewer time, CI churn, latent technical debt, slower release cycles, and erosion of trust in automation. These costs can outweigh nominal speedup gains from rare correct optimizations.
  • The finding that lighter/cheaper models can be better calibrated (and thus cheaper in total cost of ownership for certain pipelines) suggests procurement should consider calibration-adjusted cost rather than raw model size or headline capability.
• Product & Market Effects
  • There is room for new market services: calibration evaluation suites, runtime-verification integration (to ensure claimed speedups actually materialize), and certification services for abstention/behavioral calibration.
  • Model pricing and tiering may shift: buyers may prefer smaller, well-calibrated models for localized optimization tasks and reserve larger models for exploration/agentic search where hallucination risk is acceptable and verification pipelines exist.
• Deployment Strategies & Risk Management
  • Low-cost, inference-time guardrails (penalty prompts) are an immediate mitigation that preserves capability while reducing over-edits; however, they are not fully sufficient—residual over-edit rates persist and vary by task.
  • Runtime verification (measuring execution speed of candidate edits) and CI-level performance checks should be standard for production deployment of optimization agents to convert probabilistic abstention signals into deterministic guarantees.
  • Procurement contracts and internal policies should account for calibration trade-offs: require zero or low false abstention and quantify acceptable bluff rates.
• Training & R&D Incentives
  • There is an economic case for training/fine-tuning objectives that penalize over-editing or explicitly teach abstention (e.g., RLHF with abstention utility), which could become a competitive differentiator.
  • Benchmarks must evolve: include Optimal Baselines, abstention targets, and cost-aware utility functions to reflect real-world economic harms of hallucinated edits.
• Policy & Liability
  • For organizations with regulatory or safety exposure, model calibration metrics could inform audit trails and contractual liability; insurers and regulators may start to require calibration evidence for automated code-change systems.

Limitations and caveats relevant for economic decisions - This is a controlled, small-scale pilot (180 trials, 5 problems); broader datasets, agentic systems, and full-codebase evaluations may yield different bluff dynamics. - The IIV penalty is inference-only and does not eliminate hallucinations—runtime verification and pipeline safeguards are necessary for high-assurance deployment. - Model behavior may change with newer architectures, fine-tuning for abstention, or with agentic wrappers that include search and verification loops; economic trade-offs should be revisited as tooling evolves.

Bottom line for practitioners and purchasers: measure and price calibration, not just raw capability. Simple, low-cost guardrails reduce hallucination-induced operational costs substantially, but lasting economic solutions require benchmark redesign, runtime verification, and possibly new model training objectives that internalize the cost of false positive optimizations.