Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

Agentic coding assistants are increasingly given extra capabilities, such as browser based testing tools and design oriented system prompts, on the assumption that more capability yields better software. This study tested that assumption directly. Ninety independent agent runs built the same application, a real time retrospective board, from one detailed specification, each scored on a fixed 14 criterion functional rubric (42 point maximum) and a visual quality review. The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. A criterion level analysis revealed what run totals conceal. Container deployment was the dominant defect, failing first try in 44 percent of runs, with its failure rate shifting sharply across model generations while mean totals moved less than a point. The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. The practical lesson is to match the fix to the failure: most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch.

Summary

Main Finding

Across 90 matched agent runs building the same web app, model capability and reasoning effort—not adding a browser-based testing tool—most reliably produced first-try correctness. Frontier models clustered near the 42/42 functional ceiling; raising reasoning effort (High → xHigh) sharply improved first-try perfection (28% → 89%) and cut corrective prompts by ~5× for a modest cost increase (reported session-cost premium ≈ 9–29%). By contrast, enabling a Playwright-style testing automation tool raised median session cost (≈42–68%) without improving functional score or first-try reliability. A short, one-paragraph design directive reproduced the visual-quality gains of a longer commercial design prompt.

Key Points

Capability tier dominates: Claude Opus 4.7/4.6 and Sonnet 4.6 mean ≈ 41/42; a low-cost local model (Qwen) scored far lower (24 and 37) and consumed many more tokens/iterations.
First-try reliability differs inside the frontier: aggregate post-repair totals hide substantial differences in what runs get right on the first attempt.
Reasoning effort (High → xHigh) is highly effective:
- First-try perfect runs rose from 28% to 89% (abstract / paper summary).
- Corrective prompts per run fell roughly fivefold.
- Median cost rose modestly (paper reports per-cell medians implying ≈9–29% extra).
Testing tool (Playwright) increased cost without functional benefit:
- Median session cost rose 42% at High effort and 68% at xHigh for Opus 4.7.
- Functional scores were essentially unchanged.
- Token-profile shows cost increased via cache-read/context tokens (re-reading inflated session context/screenshots), not additional useful engineering output.
Design prompt effects:
- Raised visual-quality rating (mean 4.5 vs 3.0 on a 1–5 scale) without increasing functional score.
- A short (one-paragraph) paraphrase reproduced the visual lift—invocation, not verbosity/tone, drove the effect.
Failure modes concentrated in environment/build criteria:
- Docker deployment failed first-try in 40/90 runs (44%).
- Local dev environment failures also common; environment issues together accounted for ≈55% of first-try criterion failures.
- Many failures are outside what a page-level testing tool can observe (build/config/persistence).
Cost accounting nuance: the study used token-based /cost estimates (hosted-model pricing); local-run cost estimates (Qwen) are token-pricing proxies, not market prices.

Data & Methods

Task: single detailed OpenSpec change to implement a real-time retrospective board (React + Vite frontend, Node backend, SQLite persistence, WebSockets, Docker single-container deployment, CSV export, docs).
Dataset: 90 graded runs across model families: Claude Opus 4.7 (48), Opus 4.6 (27), Sonnet 4.6 (10), Gemini 3.1 (3), Qwen (2). Most controlled contrasts run on Claude Code.
Experimental factors (within Claude Code/Opus 4.7 grid): reasoning effort (High vs xHigh), testing automation tool (Playwright on/off), design prompt (none / full third‑party / abridged one-paragraph). Opus 4.7 sweep: 2 (effort) × 4 (tool/prompt) grid, 6 repeats per cell.
Scoring:
- 14 functional criteria, each scored 3 = passed first try, 2 = failed then fixed after exactly one corrective prompt, 1 = unresolved after repeated corrective prompts (max total = 42).
- Corrective prompts per run = count of criteria scored 2 (lower bound on human interventions).
- Visual quality rated 1–5 from screenshots by an LLM rater, verified by a human.
Cost & logs: session /cost (token-based estimate), token breakdown (input/output/cache-read), model time, lines of code added. Analyses descriptive; Fisher exact tests used for some contrasts; study observational (matched contrasts, not randomized).
Data availability: per-run rubrics and the dataset are openly available per paper.

Implications for AI Economics

Spend where it reduces human labor, not where it merely increases token consumption:
- Investing compute (higher reasoning effort or stronger model family) produced large reductions in human corrective interventions and much higher first-try reliability for modest incremental token cost. This converts directly to lower human oversight/repair costs.
- By contrast, adding a page-level testing tool increased token consumption (and hence billing) substantially but did not reduce repair burden because most failures were build/runtime/environment issues outside the tool’s observability.
Beware of false economies from cheap local models:
- Low-cost local models may appear cheaper per-inference but can require many more iterations and human fixes, driving up total project cost (token-and-labor combined). Procurement decisions should account for expected repair burden, not just per-call price.
Tooling costs may be driven by context re-reading:
- Tools that add roundtrips, screenshots, or large context expansion can inflate cache-read tokens; their value must be justified by catching failure modes that actually dominate repair burden.
Prompting is cheap and effective for certain objectives:
- Short, well-targeted instructions (e.g., a one-paragraph design directive) can deliver most of the benefit of longer, commercial prompts—low marginal cost to replicate across projects.
Metrics to track for economic decision-making:
- First-try correctness and corrective-prompt counts are more economically meaningful than post-repair aggregate scores; they map to human labor and calendar time.
- Token breakdown (output vs cache-read) clarifies whether added cost is buying engineering output or redundant context re-ingestion.
Procurement & team practice recommendations:
- Prioritize stronger models or higher effort settings when the goal is to reduce human verification and rework.
- Reserve investment in page-level automated testing tools for contexts where build/runtime/configuration failures are already controlled and most failures are truly UI-visible.
- Include repair-burden and first-try reliability in cost-benefit analyses of agentic tooling choices.
Limitations for economic generalization:
- Single task and environment; observational design; small sample sizes for some model families (Gemini, Qwen). Local-model cost estimates are token-priced proxies. Results should be validated across tasks and teams before broad procurement policy changes.

If you want, I can: - Produce a one-page decision checklist for managers choosing between buying stronger models, enabling higher-effort settings, or adding UI testing tooling. - Convert the key numeric contrasts into a simple ROI-style table (expected reduction in corrective prompts vs incremental token cost) for budget planning.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The study uses a controlled experimental design with multiple treatments and objective rubriced outcomes, giving reasonably persuasive within-task causal evidence about how model capability and effort affect build success; however external validity is limited to one application, a modest sample (90 runs), and possible scoring/operationalization biases, so strength is not high. Methods Rigormedium — Design includes multiple factors, a fixed multi-criterion rubric, and clear outcome measures (functional score, deployment success, visual rating), but the paper appears to rely on a single task, lacks detail about randomization/blinding and coder reliability, and does not report power calculations or robustness checks across diverse tasks/models, which limits methodological rigor. SampleN = 90 independent agent runs all tasked to build the same real-time retrospective board from one detailed specification; runs spanned several model generations (frontier vs lower-cost local model), two agent harnesses, two reasoning-effort settings (High vs xHigh), presence/absence of a browser-based testing tool, and two design-oriented system prompts; outcomes: 14-criterion functional rubric (0–42), visual quality rating (1–5), deployment/first-try success and corrective prompts/cost metrics. Themesproductivity human_ai_collab adoption IdentificationControlled factorial experiment: 90 independent agent runs building the same application with systematic variation across model generation (capability tier), two agent harnesses, two reasoning effort levels, presence/absence of a testing tool, and two design-oriented prompts; causal effects inferred by comparing outcome distributions across these manipulated conditions using a fixed 14-criterion functional rubric and visual quality ratings. GeneralizabilitySingle application (retrospective board) — may not generalize to other software types or complexity levels, Specific models, harnesses, and tool implementations — results may change with other architectures or newer model versions, Laboratory-style runs do not capture team workflows, long-term maintenance, or integration costs, Scoring subjectivity and lack of reported blinding or inter-rater reliability limit transferability, Moderate sample size and possibly unbalanced factor combinations reduce external validity

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
Ninety independent agent runs built the same application from one detailed specification, each scored on a fixed 14-criterion functional rubric (42 point maximum) and a visual quality review. Output Quality	null_result	functional score (14-criterion rubric) and visual quality review	Reading fidelity high Study strength high	n=90 0.8
The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Other	null_result	experimental condition coverage (model generation, harness, effort level, testing tool, prompts)	Reading fidelity high Study strength medium	n=90 0.48
Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. Output Quality	positive	functional score (rubric total out of 42)	Reading fidelity high Study strength medium	n=90 24 to 37 points 0.48
Container deployment was the dominant defect, failing first try in 44 percent of runs. Error Rate	negative	first-try container deployment failure rate	Reading fidelity high Study strength high	n=90 44% 0.8
The container deployment failure rate shifted sharply across model generations while mean totals moved less than a point. Output Quality	mixed	container deployment failure rate by model generation; mean functional score change	Reading fidelity medium Study strength medium	n=90 mean totals moved less than a point 0.29
The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Organizational Efficiency	negative	development cost, functional score, reliability (interface-visible criteria)	Reading fidelity high Study strength medium	n=90 42 to 68 percent 0.48
Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. Error Rate	positive	first-try perfect run rate, number of corrective prompts, development cost	Reading fidelity high Study strength high	n=90 28% to 89% (first-try perfect runs); five fold reduction in corrective prompts; 9 to 29 percent more cost 0.8
A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. Output Quality	positive	visual quality score (5-point scale) and functional score	Reading fidelity high Study strength medium	n=90 4.5 versus 3.0 on a 5 point scale 0.48
Most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch. Error Rate	positive	source of first-run failures (reasoning vs visible/interface flaws) and mitigation effectiveness	Reading fidelity medium Study strength medium	n=90 0.29

Frontier coding models and greater reasoning effort sharply raise first-run success and cut correction needs; a browser-based testing tool increased costs without improving functional outcomes, while a design prompt improved interface aesthetics but not functionality.