0 cumulative citations
View corpus contextReliability of AI coding agents hinges on the engineering around the model, not just model capability; the monograph catalogs 206 practices, measurement protocols and failure analyses showing that retrieval, state and execution infrastructure often cause apparent failures and must be tested and hardened for deployable productivity gains.
Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model
This engineering monograph argues that coding-agent reliability is primarily a system-level property—many apparent model errors arise from retrieval, execution, state management, and verification layers—and provides a 206-item practice catalog, evidence maps, and runnable protocols to evaluate and operate reliable agent systems.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
AI coding agents are commonly evaluated as models but deployed as systems. Their reliability depends not only on model capability, but on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. This monograph examines those boundaries and develops a framework for evaluating and operating coding agents reliably. It synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis. Across this evidence, many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes. Evaluation and operation are treated as a dependency chain in which weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions. The monograph contributes a versioned catalog of 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads; an evidence ledger; a framework for dependency and repair asymmetry across the agent lifecycle; measurements and failure cases from operated agent systems; runnable evaluation and reliability protocols; and five reusable agent skills with evidence maps. Together, these provide a system-level methodology for distinguishing model capability from infrastructure effects, designing defensible evaluations, and building systems that recover safely when components fail. The review is structured rather than exhaustive, evidence strength varies by topic, and results depend on workload and configuration. The methods record which search lanes were executed, which remain unexecuted, and limits on evidence-grading claims.
Summary
Main Finding
AI coding-agent reliability is a system property, not just a model property. Many observed failures attributed to “the model” actually arise in surrounding infrastructure (harness, execution and state management, retrieval, memory, permissions, review interfaces, resource allocation). Improvements measured at a single layer often fail to propagate end-to-end. To evaluate and operate coding agents defensibly you must treat evaluation and operation as a dependency chain and instrument, test, and harden each layer so that claims about model capability are not confounded by infrastructure effects.
Key Points
-
Dependency-chain perspective
- Reliability depends on measurement → grading → containment/recovery → retrieval/context → review/accountability → allocation/cost.
- Each layer supplies evidence the next layer relies on; a failure upstream can invalidate downstream conclusions.
-
Empirical synthesis and artifacts
- Versioned catalog of 206 reliability records (193 gated practices, 56 treated in depth, 13 research leads).
- Evidence ledger linking claims to sources; runnable evaluation and reliability protocols; five reusable agent skills with evidence maps.
- Synthesized evidence: 164 scholarly works, 100 practitioner records, 29 benchmark records, 17 author-system case records.
-
Evaluation and measurement guidance
- Account for run-to-run variance; use paired comparisons and power analysis (one run = one draw).
- Prefer baselines and ablations (remove component and rerun) and jointly consider cost–accuracy tradeoffs.
- Beware benchmark contamination and oracle strength; measure public–private gaps and keep task windows ahead of models.
-
Execution-based grading and gates
- Let execution/sandbox outcomes determine whether artifacts move forward (correction gates, release tests).
- Design verifiers as measured systems: separate verifier infrastructure health from oracle adequacy; bind verdicts to artifact and environment identities.
-
Containment, durable execution, and recovery
- Treat the software factory as a distributed system with explicit lifecycle contracts (logical work vs attempts).
- Use durable intent/operation identities, epoch fencing, idempotent retries, and recovery paths outside the failure domain.
- Employ replayable traces and fault-injection tests to measure recovery guarantees and ambiguous-commitment behavior.
-
Context engineering: retrieval, budgets, memory
- Score retrieval and generation separately; measure retrieval metrics and run retrieval protocols on the operated workload.
- Use localization funnels and typed indexes cautiously (only when maintainable).
- Design usable context budgets (what remains useful) and move bulk outputs out of active context; build memory only where measured retrieval failures justify it.
-
Human review and accountability
- Make correctness cheap to challenge; route escalations where they can change outcomes; widen autonomy one action class at a time.
- Put provenance where review happens and test that gates actually change execution.
- Auditability and human-auditable failure taxonomy are crucial for causal assignment and operating records.
-
Cost, topology, and allocation research agenda
- Consider agent topology, fan-out vs single-agent, and which work is eligible to dispatch.
- Cost-aware fleet scheduling, admission controls, reserved recovery capacity, and allocation ledgers determine marginal costs and reliability under load.
-
Practical patterns and failure modes
- Many failure classes and recovery contracts are enumerated; the monograph provides runnable protocols and synthetic failure-case measurements (e.g., across 1,750 trajectories one model produced submissions on 100% of trials but resolved only 44% under an external oracle).
-
Limitations
- Review is structured but not exhaustive; evidence strength varies by topic; empirical outcomes depend on workload and system configuration. The methods section documents search lanes performed and unperformed.
Data & Methods
-
Evidence synthesis
- Structured multivocal review combining academic literature, practitioner records, benchmarks, and author-system case records.
- Targeted update audits, a software-engineering coverage probe, and a distributed-systems evidence synthesis.
-
Quantities and artifacts
- 164 scholarly works, 100 practitioner records, 29 benchmark records, 17 author-system case records.
- A catalog of 206 reliability records, an evidence ledger linking claims to sources, runnable evaluation protocols, and sample operational measurements.
-
Methodological practices emphasized
- Paired, repeated trials; power analysis and paired item-by-item comparisons.
- Baselines/ablations and cost–accuracy frontiers for fair comparisons.
- Execution-based assessments (sandboxed runs, immutable verdicts), independent verification, replay and fault injection, taxonomy-driven failure analysis.
- Distinguishing retrieval vs generation performance and measuring retrieval effectiveness on the actual operated workload.
-
Reproducibility and transparency
- Companion repository with practice catalog and evidence; methods section documents which search lanes were executed and which were omitted, noting how omissions limit evidence grading.
Implications for AI Economics
-
True productivity impact is lower than naive model-only metrics
- Economic valuations based on model capabilities (benchmarks) will overestimate expected gains unless they account for end-to-end success rates, verification costs, retries, and containment overheads.
- Example driver: a model that “submits” code frequently but only resolves when verified reduces realized automation value and increases human-completion costs.
-
Hidden operational costs matter
- Retrieval/indexing, context budgets, memory infrastructure, human-review and escalation capacity, auditability/provenance systems, and recovery capacity all add both opex and capex that must be included in ROI and TCO calculations.
-
Benchmark inflation and discounting
- Public benchmark scores are subject to contamination and oracle weaknesses; economists should discount public numbers or prefer execution-based, workload-aligned measures when computing expected labor substitution or productivity improvements.
-
Cost–accuracy frontiers and allocation decisions
- Practical deployments must optimize along cost–accuracy frontiers, not only maximize accuracy: model routing, fleet scheduling, and topology choices (fan-out, ensembles, persistent agents) affect marginal cost per task and failure externalities.
-
Risk and liability economics
- Provenance, auditability, and effective human gates change the liability calculus and insurance/contracting costs. Systems that cannot provide auditable recovery and provenance will face higher regulatory and contractual frictions.
-
Investment and research priorities
- Investments in systems engineering (containment, verifiers, replay/fault-injection, retrieval measurement) can yield larger reliability returns than marginal model improvements.
- Research and product teams should prioritize measurement that isolates model capability from infrastructure effects to avoid misallocated R&D spend.
-
Metrics for economic models
- Incorporate: end-to-end resolution probability; average retries per task; verification time and human-review rate; retrieval miss rate and associated remediation; recovery-cost probability and mean loss; context budget limits (affecting task size and throughput).
- Use these to adjust forecasts of labor substitution, throughput, and unit cost.
Overall, the monograph reframes coding-agent economics: to predict value, costs, and risks you must account for the whole system around the model—measurement, execution, recovery, context, human reviews, and allocation policies—rather than extrapolating from isolated model benchmarks.
Assessment
Paper Typereview_meta
Evidence Strengthmedium — Structured multivocal review synthesizing 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records plus operated-system measurements gives broad, practice-grounded evidence, but the sources are heterogeneous, selection and publication/practitioner biases are possible, and claims are workload- and configuration-dependent rather than supported by strong causal identification.
Methods Rigormedium — The monograph documents search lanes, an evidence ledger, targeted audits, coverage probes, and runnable protocols, showing careful engineering and transparent limits; however it is not a fully exhaustive systematic review, some search lanes are unexecuted, much evidence is observational/practitioner-sourced, and formal causal or statistical identification is limited.
SampleMultimodal evidence synthesis: 164 scholarly publications, 100 practitioner records, 29 benchmark records, 17 author-system case records, plus empirical measurements from operated agent systems (e.g., reported 1,750 trajectories over ~50 coding tasks and four models in examples), and a versioned catalog of 206 reliability practices/records.
Themeshuman_ai_collab productivity org_design
GeneralizabilityFindings and practices are specific to coding agents and software-engineering workflows and may not transfer to non-coding agent domains (e.g., medical, legal) without adaptation., Empirical results depend on workload mix, system configuration, and chosen models; observed failure rates and effectiveness of controls may not generalize across organizations., Practitioner records and author case studies may suffer from selection bias and limited representativeness., Rapid model and tooling changes mean some practices or measurements may age quickly (time sensitivity)., Benchmarks and public scores may be contaminated or not reflect deployed workloads, limiting external validity.
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Coding-agent reliability depends on system components surrounding the model, including the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation, rather than on model capability alone. Organizational Efficiency | mixed | End-to-end coding-agent reliability |
Reading fidelity
high
Study strength
medium
|
n=310
|
| Many apparent coding-agent model failures originate elsewhere in the surrounding system, and improvements measured at one system layer often do not propagate to end-to-end task outcomes. Output Quality | mixed | End-to-end task outcomes and failure attribution |
Reading fidelity
high
Study strength
medium
|
n=310
|
| In an evaluation covering 1,750 trajectories across 50 tasks and four models, one model submitted work in every trial but resolved only 44% of trials under an external oracle. Output Quality | negative | Successful resolution of coding tasks after submission |
Reading fidelity
high
Study strength
medium
|
n=1750
100% submitted; 44% resolved
|
| A passing benchmark score supports a coding-agent capability claim only when the benchmark matches the target workload, the solution is independent of prior or in-task exposure, and the oracle rejects incorrect artifacts. Output Quality | positive | Validity of benchmark-based capability conclusions |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Candidate-file selection creates an upper bound on retrieval recall because any file omitted at that stage remains unavailable to later inspection and editing stages. Output Quality | negative | Repository evidence recall during coding-agent localization |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Only independently observed evidence, rather than a worker's self-report, should be allowed to complete logical work in the agent workflow. Organizational Efficiency | positive | Correct completion-state assignment for logical work |
Reading fidelity
high
Study strength
medium
|
not reported
|
| A human review is an effective gate only when its rejection or modification changes the next reachable execution state; an advisory review that leaves execution reachable is not an effective gate. Governance And Regulation | positive | Effectiveness of human review and enforcement controls |
Reading fidelity
high
Study strength
medium
|
not reported
|