Agent-based software engineering spends most tokens refining code, not generating it: in 30 ChatDev GPT‑5 projects the Code Review stage consumed 59.4% of tokens while input tokens made up 53.9% of usage, indicating iterative agent collaboration is the primary operational cost driver.

Main Finding

Agentic software engineering (LLM-based multi-agent) token costs are highly concentrated in iterative refinement and verification rather than initial code generation. Using ChatDev with a GPT-5 reasoning model on 30 end-to-end SDLC tasks, the Code Review stage consumed an average of 59.4% of tokens per task, and input/context tokens made up the majority of consumption (avg. 53.9% of tokens). This implies the dominant economic cost of agentic SE is the “communication tax” from passing large contexts repeatedly between agents.

Key Points

• Concentration of cost:
  • Code Review = 59.4% of tokens on average (most expensive stage).
  • Coding (initial generation) = 8.6% on average; Design = 2.4% (cheap relative to refinement).
  • Code Completion (when executed, n=6) averaged 26.8% of tokens; Testing (n=12) averaged 10.3%; Documentation averaged 20.1%.
• Token-type breakdown (overall per task):
  • Input tokens: 53.9%
  • Output tokens: 24.4%
  • Reasoning tokens: 21.6%
  • Roughly a 2:1 input:output ratio, consistent with a “communication tax.”
• Phase-specific token profiles (avg % input / output / reasoning):
  • Design: 60.4 / 3.6 / 36.0
  • Coding: 6.9 / 58.0 / 35.1 (output-heavy)
  • Code Completion: 47.7 / 41.7 / 10.5
  • Code Review: 51.4 / 24.7 / 23.9 (input-heavy)
  • Testing: 60.8 / 20.7 / 18.4
  • Documentation: 80.2 / 8.3 / 11.5 (very input-heavy)
• Evidence suggests agentic verification/refinement workflows are the primary source of inefficiency and cost.

Data & Methods

• System: ChatDev multi-agent framework instrumented to log full execution traces (prompts, responses, token counts).
• Tasks: 30 diverse software development prompts from the ProgramDev dataset (simple algorithms → complex apps). Reasoning-token consumption per task ranged ~17,280 to 40,000 tokens (indicating diversity).
• Model: GPT-5 reasoning model (gpt-5-2025-08-07). Key settings: temperature fixed at 1.0, context window 400k tokens, max output tokens 128k.
• Phase mapping: Mapped ChatDev internal phases to SDLC stages to enable cross-task aggregation (Design, Coding, Code Completion, Code Review, Testing, Documentation).
• Analysis pipeline: parse execution traces → map phases to SDLC stages → aggregate token counts by stage and by token type (input/output/reasoning) → compute averages and variability across 30 runs.
• Limitations noted by authors: single LLM and single LLM-MA framework; some phases occurred only in a subset of runs (e.g., Code Completion n=6, Testing n=12); mapping abstraction may not generalize perfectly to other frameworks.

Implications for AI Economics

• Direct financial impact:
  • Since providers charge per token, the majority of runtime cost for agentic SE workflows will accrue during iterative review/verification, not initial code generation. Project costing should weight refinement-heavy tasks more heavily.
• Energy & carbon consequences:
  • High token counts (especially repeated transmission of large contexts) imply higher compute and energy usage—refinement cycles drive most of the environmental footprint of agentic SE.
• Predictability & budgeting:
  • Tokenomic profiles create a “cost map” by task type: greenfield coding vs. refactoring/verification have very different cost structures. This enables better budgeting, pricing, and cost–benefit analyses for automation vs. human labor.
• Optimization opportunities (practical levers to reduce cost):
  • Reduce context passing: send deltas or compressed summaries instead of full code each round.
  • Mixed-model pipelines: use smaller/cheaper models for reviewers or for parts of the review loop; reserve large reasoning models for key decisions.
  • Cache and reference artifacts via embeddings/vector stores rather than resending raw code.
  • Adaptive gating: introduce human-in-the-loop checkpoints before expensive agentic review loops to avoid needless iterations.
  • Structured verification protocols: design verification primitives (e.g., targeted tests, patch-only reviews) that avoid full-context dialogues.
  • Instrumentation & budgeting tools: integrate token accounting into CI/CD to predict and cap costs per PR/task.
• Research & market implications:
  • Need for standardized benchmarks and token-efficiency metrics for LLM-MA architectures to compare economic tradeoffs (not just task accuracy).
  • Model/architecture heterogeneity matters: tokenomics should be measured across different LLMs and multi-agent protocols before generalizing pricing assumptions.
  • Opportunity for differentiated service offerings: “refinement-optimized” agent stacks (lower-cost review pipelines) could be a competitive product niche.
• Caveats:
  • Results are preliminary and system/model-specific; monetary or carbon conversions require local provider rates and power-emissions factors.
  • Small-sample occurrences of some phases mean per-phase estimates (e.g., Code Completion) need more data to become stable.

Overall takeaway for AI economics: when evaluating the cost-effectiveness of agentic software engineering automation, account for refinement/verification as the dominant operational expense and prioritize protocol and architecture changes that reduce repeated full-context communication.