← Papers

LLM agents decay with task length: success falls roughly geometrically with each dependent step, and even state-of-the-art models collapse from near-perfect to near-zero on agentic tool-use within sixteen steps; trimming context windows does not prevent — and can worsen — the decline.

How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making

Shubhra Mittal · August 31, 2026

arxiv quasi_experimental high evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: September 3, 2026

Shubhra Mittal unresolved corpus identity

Semantic Scholar

Latest observation: September 3, 2026

Shubhra Mittal provider ID

Across nine models and four synthetic task families, per-step reliability governs a near-geometric decline in multi-step agent success so that even top models collapse to very low end-to-end success at long horizons (agentic tasks often fail within ~16 steps).

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed September 3, 2026

View corpus context

Production deployments of large language model (LLM) agents remain unreliable on long, multi-step workflows even as benchmark success rates climb steadily. We argue this gap is largely an artifact of task horizon: benchmarks are dominated by short-to-medium horizons where success remains high, while production workloads demand an order of magnitude more dependent steps. We measure the effect directly, characterizing the shape of agent degradation and disentangling its cause across a large controlled study spanning nine models, six open models from 1.2B to 671B parameters, and three deployed proprietary systems; four task families, including a genuinely agentic tool-use loop; five horizons; and three context regimes. Task success follows a geometric law governed by a single per-step reliability parameter, which rises with model scale but saturates well below 1 even for the strongest models, guaranteeing eventual collapse at sufficiently long horizons. The effect is sharpest on the agentic task, where every model tested, including widely deployed systems, falls from near-perfect success to near zero within sixteen steps of (n=10,664 analyzed trajectories. Degradation is driven by step count rather than context length: bounding the context window steepens decay rather than easing it (logit slope -0.69 vs. -0.44), p=3x10-6), contradicting a lost-in-the-middle explanation and warning against a common production shortcut. Projecting measured reliability onto representative benchmark horizons quantifies a substantial gap between benchmark and production conditions, from 0.42 at GAIA-length horizons to 0.24 at hundred-step production horizons. For teams responsible for agent orchestration and reliability at scale, these results argue for horizon-aware evaluation and reliability budgeting in place of aggregate pass-rate metrics. Code, prompts, seeds, and raw trajectories are released.

Summary

Main Finding

Task success for LLM agents across multi-step workflows decays approximately geometrically with horizon: success ≈ r^H where r is a measurable per-step reliability < 1. Per-step reliability improves with model scale but saturates well below 1, so any non-trivial agent is mathematically guaranteed to “collapse” at sufficiently long horizons. Degradation is driven primarily by step count (number of dependent actions) rather than by context length; bounding/truncating context can worsen, not help, long-horizon reliability.

Key Points

Scope and scale of study
- 9 instruction-tuned models: 6 open models (1.2B→671B params) + 3 proprietary deployed systems.
- 10,664 analyzed trajectories across four synthetic task families (three streaming-like + one genuinely agentic tool-use loop).
- Geometric decay model selected by AIC in 28 of 36 model×task cells.
Quantitative summaries
- Mean measured per-step reliability r ≈ 0.61 (author pooled mean).
- Projected success by representative horizons (using r=0.61):
- 8 steps (GAIA-like): ~0.42
- 15 steps (WebArena-like): ~0.36
- 20 steps (tau-like): ~0.33
- 30 steps (SWE-bench/OSWorld-like): ~0.30
- 100 steps (production-like): ~0.24
- Agentic task (ToolQA): every model tested drops from near-perfect at H=2 to near-zero by H=16 in many cases; collapse within 16 agentic steps is typical.
- Hazard accelerates within a trajectory: mean per-step accuracy falls from 0.58 (first third) to 0.44 (last third).
- Interface/format errors (unparsable tool calls) affect ~21% of trajectories and increase with horizon.
Step-count vs. context-length disentangling
- Three regimes: natural (one instruction per turn), compressed (shortened context via carried state), padded (all operations in one turn).
- Natural regime performs best. Compressed regime has a steeper logit-slope per doubling of horizon (-0.69) than natural (-0.44); difference compressed vs. natural p = 3×10^-6.
- Collapsing all ops into a single prompt (padded) decays at about the same slope as natural but from a lower baseline.
- Conclusion: failures come from executing many dependent steps, not from “lost in the middle” long context per se. Naive context truncation is likely counterproductive.
Failure dynamics & operational observation
- Errors behave like absorbing states: once a wrong step occurs, recovery is rare, and downstream steps compound the failure silently.
- This implies end-to-end monitoring detects failures too late; step-level verification or cheap oracles can detect and stop cascading failures early.
Limitations
- Tasks are synthetic for oracle verifiability (although ToolQA is genuinely agentic).
- Nine-model sample—not exhaustive of all current/future models.
- Fixed decoding temperature; sensitivity to decoding settings not explored fully.
- API budget limited the longest horizons in the agentic family; H=16 collapse is a lower bound.
Reproducibility
- Code, prompts, seeds, raw trajectories, and analysis scripts released: https://github.com/shubmittal/agent-horizon-degradation

Data & Methods

Task families
- Ledger: sequence of account updates → arithmetic working memory.
- Refchain: track one variable through assignments + distractors → long-range retrieval.
- Cipher: ordered string edits → procedural execution, sensitive to single mistakes.
- ToolQA: agentic ReAct loop traversing hidden chain via tool calls; agent must issue inspection calls and cannot preplan full path.
Context regimes (paired design)
- Natural: operations arrive one per turn, full multi-turn history preserved.
- Compressed: same operations & turns but with shortened context (carried state, windowed history).
- Padded: all operations for a horizon delivered in a single turn.
Statistical design
- Paired comparisons across regimes (same task instance used across regimes).
- Confidence intervals via Wilson score intervals for per-cell success rates.
- Model selection among candidate decay shapes (geometric, threshold/cliff, linear) by AIC.
- Deterministic decoding (fixed temperature), seeds logged.

Implications for AI Economics

Reliability is a horizon-dependent economic resource
- Treat per-step reliability r as a key economic parameter. End-to-end success probability ≈ r^H; doubling H can sharply reduce success and thus realized value.
- Benchmarks that focus on short horizons overstate production reliability. Market signals based on aggregate pass rates will misprice the risk of long workflows.
Cost & procurement consequences
- Diminishing marginal returns to scale: larger models increase r but with saturation—buying model scale has limited upside in reaching near-perfect per-step reliability. Procurement decisions must weigh incremental cost vs. marginal improvement in r.
- For long-horizon workflows, spending on larger models alone is a weak lever; complementary investment (verification infrastructure, modularization, checkpoints) likely has higher ROI.
Pricing, SLAs, and risk allocation
- SLAs and pricing for agent-powered services must account for geometric failure risk and the expected number of dependent steps. Fixed-price long workflows imply asymmetric risk that should be priced or hedged.
- Reliability budgeting: organizations should set explicit per-step reliability targets and derive achievable horizons for pricing, capacity planning, and expected rework costs.
Product design and architecture
- Minimize dependent-step chains where possible. Decompose workflows, insert deterministic modules, or redesign interfaces to reduce H.
- Invest in cheap, automated step-level verification (oracles, checksums, unit tests) to detect failures early and avoid large downstream costs. These checks can dominate ROI compared to raw model upgrades.
- Avoid naive context truncation as a cost-control measure—truncation can steepen reliability decay and increase total rework/error costs.
Monitoring, incident response, and operational metrics
- End-to-end success is a lagging indicator for long-horizon tasks; instrument step-level metrics and early-warning signals to reduce incident cost and customer impact.
- Use per-step reliability and horizon-specific simulations to forecast expected failure rates and to budget staff/compute for retries and mitigations.
Market & benchmark design implications
- Benchmark designers and purchasers should require horizon-aware metrics (per-step reliability, decay curves) rather than aggregate pass rates.
- Vendors and third-party evaluators should report r and projected performance across a range of horizons to enable apples-to-apples economic comparisons.

Actionable checklist for decision-makers - Measure r on your real tasks (not only benchmark tasks). - Compute achievable horizon H for required end-to-end success targets: solve r^H ≥ target → H* = log(target)/log(r). - Budget for verification steps and rework; prioritize step-level checks where inexpensive oracles exist. - Re-evaluate procurement trade-offs: additional model cost vs. investment in verification/architectural changes. - Adjust SLAs and pricing to account for geometric failure risk and the cost of retries.

If you want, I can: - compute example ROI trade-offs (model-upgrade vs. verification-cost) using your cost and target-success numbers; or - translate these results into recommended SLA clauses or procurement scoring templates tailored to an enterprise use case.

Assessment

Paper Typequasi_experimental Evidence Strengthhigh — Large, reproducible dataset (10,664 trajectories), multiple model families (six open models spanning 1.2B–671B parameters plus three proprietary systems), controlled paired design that holds task instances constant across regimes, clear statistical procedure (AIC, paired comparisons, Wilson intervals), and public release of code/data enabling verification; main limitations are synthetic tasks and limited external ecological validation. Methods Rigorhigh — Rigorous, pre-registered-style controls: paired regime comparisons on identical task instances, multiple functional-form fits compared by AIC, appropriate confidence-interval methods for extreme proportions, fixed seeds/temperatures, and full data+code release; weaknesses include synthetic oracle tasks (ecological validity), limited model sample (9 models), and constrained decoding settings. SampleEvaluation of nine instruction-tuned models (six open models from 1.2B to 671B parameters and three proprietary hosted systems) on four synthetic, oracle-verifiable task families (Ledger, Refchain, Cipher, ToolQA), across five horizons, three context regimes (natural, compressed, padded), fixed decoding temperature and recorded seeds, producing 10,664 analyzed trajectories; analyses pooled streaming families and separately report the agentic ToolQA family. Themesproductivity org_design IdentificationControlled within-subject experimental evaluation manipulating task horizon and context regime across paired task instances; model selection (AIC) used to choose functional form (geometric vs threshold vs linear); paired difference tests and Wilson intervals for regime comparisons; decomposition of effects via three context regimes (natural, compressed, padded) isolates step count from context length. GeneralizabilityTasks are synthetic and oracle-verified; results may not fully generalize to messy, open-ended real-world workflows involving humans or noisy external tools., Model sample (nine models) cannot guarantee that future or other current models will follow identical decay behavior., Decoding parameters fixed (temperature moderate); sensitivity to temperature, sampling strategies, or other inference-time engineering not explored., Agentic longest-horizon conditions were limited by API budget (H=16 reported is a lower bound for collapse severity)., Proprietary models accessed via hosted interfaces may have deployment-specific wrappers or safety layers that differ from research-access models.

Claims (12)

Claim	Direction	Outcome	Confidence & Evidence	Details
Task success follows a geometric decay law in 28 of the 36 model-task cells tested. Output Quality	negative	Task success as a function of task horizon	Reading fidelity high Study strength medium	n=36 28 of 36 cells 0.48
Success declines monotonically with horizon whenever the task is non-trivial for the model. Output Quality	negative	Task success rate	Reading fidelity high Study strength medium	n=10664 0.48
On the agentic ToolQA task, every model tested collapsed to a small fraction of its starting performance by horizon 16. Output Quality	negative	ToolQA task success rate	Reading fidelity high Study strength medium	n=9 all 9 models declined to a small fraction of starting performance by H=16 0.48
Qwen2.5-72B's ToolQA success fell from 1.00 at horizons 2 and 4 to 0.13 at horizon 16. Output Quality	negative	ToolQA task success rate	Reading fidelity high Study strength medium	1.00 to 0.13 success rate 0.48
Per-step reliability increases with model parameter count across the open-model ladder. Output Quality	positive	Per-step reliability	Reading fidelity high Study strength low	n=6 Pearson r=+0.36 0.24
No model reached perfect per-step reliability on any non-trivial task. Error Rate	negative	Per-step reliability	Reading fidelity high Study strength medium	n=9 0.48
Per-step accuracy declined from 0.58 in the first third of long-horizon trajectories to 0.44 in the last third. Error Rate	negative	Mean per-step accuracy over a trajectory	Reading fidelity high Study strength medium	n=10664 0.58 to 0.44 0.48
The natural context regime performed better than both the compressed and padded regimes at every tested horizon. Output Quality	positive	Pooled task success across context regimes	Reading fidelity high Study strength medium	n=10664 0.48
Compressed context produced a steeper decline in success than the natural regime, with logit slope -0.69 versus -0.44 per horizon doubling. Output Quality	negative	Task success as horizon increases under different context regimes	Reading fidelity high Study strength high	n=10664 logit slope -0.69 vs. -0.44; p=3x10-6 0.8
The padded regime had a decay rate statistically indistinguishable from the natural regime. Output Quality	null_result	Task success decay rate	Reading fidelity high Study strength high	n=10664 logit slope -0.40 vs. -0.44; p=0.51 0.8
Format and tool-call drift affected 21% of trajectories and increased with horizon. Error Rate	negative	Format and tool-call error occurrence	Reading fidelity high Study strength medium	n=10664 21% of trajectories 0.48
Using mean measured per-step reliability of 0.61, projected task success declined from 0.42 at an 8-step horizon to 0.24 at a 100-step horizon. Output Quality	negative	Projected task success at representative horizons	Reading fidelity high Study strength low	n=10664 0.42 at 8 steps; 0.24 at 100 steps 0.24