Agentic bug-fixing frameworks burn through far more energy than they produce in value when paired with small language models; framework architecture drives energy use but limited SLM reasoning yields near-zero success, making current designs unsuitable for low-resource deployments.

Main Finding

When constrained to Small Language Models (SLMs) on local hardware, state-of-the-art agentic issue-resolution frameworks consume substantial energy while achieving near-zero task resolution. Framework architecture—not just the SLM—drives most of the energy cost, and much of that energy is effectively wasted on unproductive multi-turn reasoning loops. The study finds a severe trade-off: the only non-zero success (AutoCodeRover + Qwen-3 1.7B, ~4% resolution) came with the highest energy consumption (~208–216 kJ/run), while the most energy-efficient framework (OpenHands) resolved 0% of tasks and used ~23 kJ/run (≈9.4× less than AutoCodeRover with Gemma).

Key Points

• Frameworks evaluated: AutoCodeRover, SWE-Agent, OpenHands, Mini SWE Agent.
• SLMs evaluated: Gemma-3 4B and Qwen-3 1.7B (both run locally, FP16).
• Benchmark: SWE-bench Verified Mini (50 tasks); experiments: 1,200 runs total (3 repetitions × 50 tasks × 4 frameworks × 2 models).
• Metrics measured: total energy (CPU+GPU via RAPL and NVML), wall-clock time, token usage, LLM call count, peak RAM/VRAM, cost (per-token pricing), and resolution status (SWE-bench pass/fail).
• Core quantitative findings:
  • Energy: AutoCodeRover mean ≈ 208–216 kJ/run; OpenHands mean ≈ 23 kJ/run. AutoCodeRover used up to ~9.4× the energy of OpenHands (Gemma).
  • Effectiveness: nearly all framework+SLM configurations had 0% resolution; only AutoCodeRover+Qwen resolved 2/50 tasks on average (~4%).
  • Correlations: total energy strongly correlated with wall-clock time (R≈0.89) and output tokens (R≈0.88), indicating “chatty” multi-turn behavior drives energy.
  • Failure modes: ReAct-style agents suffered repetitive step loops and context-loss; deliberative pipelines (AutoCodeRover) still looped and timed out but were more likely to occasionally succeed—at high cost.
• Key diagnosis: limited SLM reasoning capacity is the bottleneck for success; framework orchestration and design determine how much energy that limited reasoning burns.

Data & Methods

• Hardware: Intel Xeon w3-2435 CPU, 32 GB RAM, NVIDIA RTX A2000 16 GB GPU. Idle draw measured and subtracted (51.69 W CPU, 2.70 W GPU).
• Model loading: FP16, 32K context window; peak VRAM observed ~11–12 GB (Gemma) and ~8–9 GB (Qwen).
• Experimental controls:
  • Isolation per run, 30-minute wall-clock timeout, consistent software harness (SWE-bench evaluation).
  • Instrumentation: RAPL for CPU energy, NVML for GPU energy, OS counters for memory, token counters for LLM usage.
  • Repetition: 3 full repetitions per task/configuration → 150 runs per condition.
  • Qualitative log analysis on a subset (200 runs) to categorize failure modes using an existing taxonomy (MAST).
• Outputs provided: raw logs, JSON metrics, and replication package (paper repository).

Implications for AI Economics

• Cost-effectiveness and ROI
  • Very low success rates combined with substantial energy use imply very poor cost-per-success for SLM-driven agentic workflows as currently architected. Resource-limited deployments (edge/consumer hardware) will see low ROI unless frameworks and models are adapted.
  • Energy is a direct operational cost and an environmental externality. High per-run energy with near-zero resolution multiplies both financial and carbon costs for any attempted automation scale-up.
• Infrastructure trade-offs: local versus cloud
  • SLMs enable local inference and lower absolute per-inference compute requirements, but framework-induced multi-turn overhead can negate those savings. In many cases, cloud-hosted LLMs (or hybrid strategies) may still be economically preferable if they reduce iterations or improve success probability.
• Investment priorities
  • Investing solely in smaller models is insufficient. Greater economic value is likely realized by redesigning orchestration architectures to reduce redundant calls, token churn, and reasoning loops—i.e., engineering effort in the agent stack may outweigh marginal gains from model improvements alone.
  • Cost-aware tooling (early stopping, action pruning, verification shortcuts, caching, selective model escalation) should be prioritized in product roadmaps to achieve acceptable economic returns.
• Policy and procurement
  • Procurement and sustainability assessments for AI tooling should include joint measures of effectiveness and energy (energy-per-success) rather than raw compute or model size.
  • When evaluating autonomous developer tooling, buyers should request energy-aware benchmarks and failure-mode analyses, not just aggregated accuracy metrics.
• Research & product directions with economic impact
  • Design low-overhead frameworks that actively manage SLM limitations (adaptive orchestration, task decomposition with specialized micro-models, hybrid SLM+LLM escalation policies).
  • Develop energy-aware objective functions and monitoring (optimize for cost-per-resolved-issue).
  • Explore hardware-aware scheduling (e.g., batch verification steps, token-budget constraints) to reduce runtime and token emissions.
  • Consider mixed deployments: use cheap local SLMs for lightweight tasks and fall back to stronger cloud LLMs only when local agents signal uncertainty—this conditional-cost model can preserve local responsiveness while controlling expected costs.
• High-level takeaway for AI economics: sustainable, cost-effective autonomous developer agents require co-design of models and orchestration. Without architectural changes that limit unproductive multi-turn reasoning, switching to SLMs alone does not guarantee economic or environmental benefits.

If you want, I can: - compute approximate cost-per-run and cost-per-success using the per-token pricing and token counts from the runs, - propose a prioritized list of concrete engineering changes to minimize energy waste in agent frameworks, or - draft an experiment plan to evaluate hybrid escalation (SLM local + cloud LLM fallback) for a cost-benefit analysis.