0 cumulative citations
View corpus contextStopping likely-to-fail software-agent runs early and offering their code edits to a fresh retry saves up to ~20% of inference tokens and can increase task resolution by up to ~5 percentage points on a standard SWE benchmark, while keeping monitor overhead minimal.
Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
A lightweight 0.6B monitor that predicts likely-to-fail SWE-agent trajectories and a restart policy that offers aborted runs' repository diffs to fresh same-policy retries cuts inference tokens by up to ~20% and can raise task resolution by several percentage points on the SWE-bench Verified benchmark.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion. Early termination, however, risks interrupting trajectories that would otherwise succeed; conversely, an unsuccessful trajectory may still contain useful repository edits. We present FailFast-RestartSmart, a two-stage controller for a single active trajectory. FailFast is a lightweight 0.6B monitor trained with terminal and dense fail-to-pass supervision to predict failure from observable prefixes without policy logits or hidden states. Upon an alarm, RestartSmart launches a fresh same-policy rollout without prior prompt history and offers the interrupted repository diff as an optional overlay that the agent may inspect, apply, or discard. On SWE-bench Verified, a monitor trained solely on Qwen3.6-27B trajectories transfers to three other policies, including a closed-API model, and saves 14.6%-20.4% of execution tokens at a target 5% false-positive rate; on Qwen3.6-27B, its 20.4% saving exceeds the 12.5% achieved by our per-step AgentStop adaptation. At a target 25% false-positive rate, RestartSmart raises Qwen3.6-27B resolution from 66.6% to 71.8%, whereas cold restart reaches only 66.8%. Together, these results support early termination with sequential same-policy recovery.
Summary
Main Finding
FailFast–RestartSmart is a two-stage controller that (1) predicts likely-to-fail long-form software-engineering (SWE) agent trajectories early with a lightweight, model-agnostic monitor (FailFast) and (2) restarts the same policy while optionally exposing the aborted run’s repository edits as an unforced overlay (RestartSmart). At a tight false-positive budget (5% FPR) a single 0.6B monitor trained on Qwen3.6-27B reclaims 14.6%–20.4% of execution tokens across several policy models (20.4% on Qwen3.6-27B, beating competing stop heuristics). At looser FPRs, RestartSmart converts saved compute + partial edits into higher end-to-end resolution (e.g., Qwen3.6-27B resolution rises from 66.6% → 71.8% at 25% FPR), outperforming cold restart.
Key Points
- FailFast monitor
- Lightweight Qwen3-0.6B backbone adapted with LoRA plus three linear heads (value, F2P, P2P).
- Uses only observable text (issue + window of recent 8 steps: thoughts/actions/observations and latest patch step); does not require policy logits or hidden states → enables monitoring closed-API models.
- Trained with terminal labels plus dense fail-to-pass (F2P) progress targets obtained by replaying intermediate edits and scoring them with the task evaluator.
- Deployment meta-score fuses calibrated terminal value (Platt scaling), F2P estimate, and step fraction; gating (ϕmin ≈ 20 steps) and voting (count-M or sustained-K) avoid noisy early alarms.
- Trained on Qwen3.6-27B trajectories but transfers (without weight updates) to other models including Qwen3.5-9B, Gemma4-31B, Gemini3-Flash.
- RestartSmart restart policy
- On an abort, launches a fresh same-policy rollout with no prior prompt history.
- Extracts a stable repository overlay Δ by replaying bash commands and snapshotting the cumulative diff (ignores newly created files; waits until edits settle).
- Exposes Δ as an optional git-apply–backed overlay (diff/on/off) — the restarted agent may inspect, apply, or discard it, avoiding textual anchoring while preserving recoverable progress.
- Empirical outcomes (on SWE-bench Verified)
- At 5% FPR: FailFast saves 14.6%–20.4% of total execution tokens across the four tested policies; on Qwen3.6-27B specifically saves 20.4% vs AgentStop’s 12.5% and a duration-only control’s 11.4%.
- At 10% FPR: RestartSmart improves resolution by ~3.0–4.0 percentage points for three open-weight policies, with 20.5%–36.9% net token overhead.
- At 25% FPR: Qwen3.6-27B resolution increases from 66.6% → 71.8% (+5.2 pp) with RestartSmart; cold restart under the same alarms reaches only 66.8% (+0.2 pp).
- Monitor overhead is small relative to policies (0.6B monitor is 15–52× cheaper per token than 9–31B policies); estimated inference-energy and operational-carbon savings closely track token savings.
- Baselines and comparisons
- Compared to AgentStop (policy-execution-signal based) and a Duration-only control, FailFast reclaims more compute and generalizes better across policy models.
- RestartSmart outperforms cold restart and competes with correction/feedback methods while avoiding strong-model escalation or costly in-place corrections.
Data & Methods
- Dataset and evaluation
- SWE-bench Verified tasks, 500 instances; trajectory collection: 11 independent runs (seeds 0–10).
- Split: 350 train / 50 validation / 100 test instances. Trajectories capped at 100 steps.
- Monitor training
- Trained only on Qwen3.6-27B-generated trajectories (sampled seeds 0–10).
- Input serialization: [ISSUE] + [WINDOW] (recent 8 steps + pin to latest patch-producing step).
- Auxiliary dense supervision: fail-to-pass (F2P) and pass-to-pass (P2P) progress targets computed by replaying bash commands to produce intermediate patches and scoring them with the official evaluator; F2P found useful, P2P not helpful (λp set to 0).
- Loss: BCE on terminal label + BCE on F2P progress + ranking (Bradley–Terry-style) term; LoRA adapters trained (rank 16, α=32), heads in fp32.
- Calibration: Platt scaling on value logits; per-step meta logistic regression fuses calibrated value, ˆgf (F2P), and step fraction ϕ.
- Deployment gating: skip monitor until ≥ϕmin (empirically 20 steps); alarms aggregated via voting thresholds chosen to meet a target FPR budget.
- Restart extraction
- Replay run to find first edit at/after abort tf (or most recent prior), extend until g consecutive steps without edits, take cumulative git diff over existing baseline files (exclude newly created files) as overlay Δ.
- Overlay presented as optional tool (diff/on/off) in new run’s environment; no prior prompt context is carried.
- Baselines
- AgentStop (uses token log-probs, token counts, step repetitions) adapted to per-step decisions.
- Duration control: step-count-only thresholding.
- SWE-PRM: process-reward based in-place correction (used as an external baseline).
- Cold restart: restart without providing overlay Δ.
- Metrics
- False-positive rate (FPR) budget (fraction of would-pass runs aborted) is the primary operating constraint.
- Recall (fraction of failing runs detected), Saved (fraction of total agent tokens reclaimed), Fired (percentage of test trajectories stopped), precision, and end-to-end resolution (task success rate after restart policy).
- Compute and carbon estimation
- FLOPs per token roughly 2N; monitor overhead estimated relative to policy parameter counts to show net compute/energy savings.
Implications for AI Economics
- Direct operational cost reduction
- Token-level compute savings of ~15–20% at conservative FPRs translate to proportional inference-cost and (approx.) operational-carbon reductions. For providers paying per-token compute or managing data-center energy budgets, FailFast can materially lower marginal cost per task.
- Better utilization of compute budget and capacity planning
- Reclaimed compute can be repurposed (e.g., for retries, verifiers, or serving more users), increasing throughput without proportional hardware scaling. This enables higher effective system utilization and defers capex for capacity expansion.
- Trade-offs between cost and service quality (SLAs/pricing)
- The FPR budget is an explicit knob controlling collateral damage vs savings. Product teams can tune it to balance latent revenue/quality impact (aborting some would-pass runs) versus per-request cost — enabling differentiated SLAs or pricing tiers (e.g., “low-latency / high-reliability” vs “cost-optimized” modes).
- Reduced need for policy-specific instrumentation
- Because the monitor uses only observable text, it can be deployed across closed-API models and diverse policies without white-box integration, reducing engineering and monitoring costs across multi-vendor stacks.
- Enables cheaper, same-policy recovery strategies
- RestartSmart’s optional overlay mechanism improves success rates without escalating to a stronger (costlier) model. This can lower average cost-per-successful-task compared with strategies that resample many trajectories or escalate to larger models.
- Impact on multi-sample vs single-run economics
- Fail-fast stopping and smart restart offer a middle path between single-run deployments and budget-heavy multi-sample search: saved compute from stopped bad rolls can subsidize judicious retries, possibly reducing the need to pre-commit to expensive Best-of-N sampling for many instances.
- Design considerations for productization
- Economic benefits depend on realistic false-positive tolerance and the distribution of task difficulty; careful calibration and monitoring of user-facing quality metrics are essential.
- Optional overlay design reduces risk of harmful persistence of bad edits, supporting auditability and rollback (important for liability and trust in developer-facing tools).
- Limitations and open economic questions
- Results are on SWE-bench Verified; real-world repos, private stacks, or different task mixes may change failure patterns and the monitor’s transfer behavior.
- The method excludes newly created files from overlays; some classes of partial progress may be discarded, limiting recoverability in certain tasks.
- Further economic evaluation needed for mixed strategies (e.g., combining FailFast with occasional strong-model escalation) to quantify marginal cost of improved resolution versus larger-model inference.
- Market-level effects: widespread deployment could reduce aggregate compute demand per developer-task, affecting pricing and provisioning models for LLM compute offerings.
Overall, FailFast–RestartSmart demonstrates a practical, low-overhead lever for lowering inference costs and improving effective task resolution in long agentic SWE workloads, while preserving a controllable, tunable trade-off between cost savings and quality — a construct directly relevant to the operational economics of LLM-powered developer tools and services.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides rigorous, reproducible experimental evidence on a recognized benchmark (held-out test split, cross-fit, calibration, multiple baselines, and transfer experiments), supporting claims about token savings and resolution changes for SWE agents. However, evidence is limited to a synthetic/benchmark environment, specific agent families, and simulated cost/energy estimates; it does not establish real-world productivity or economic impacts outside the evaluated tasks.
Methods Rigorhigh — Careful experimental design: explicit train/val/test partitions, out-of-fold training and calibration, operating-point selection under explicit FPR budgets, comparisons to sensible baselines and ablations (duration-only, AgentStop, cold restart, SWE-PRM), monitor transfer tests across models, and quantification of overhead. Minor gaps include benchmarking on a single suite (SWE-bench Verified), excluding newly created files from overlays, and absence of real-world deployment or user studies.
SampleExperiments on SWE-bench Verified: 500 instances with 11 independent runs (seeds 0–10) per instance; split into 350 train, 50 validation, 100 test instances. Trajectories limited to 100 steps. Policy models evaluated include Qwen3.5-9B, Qwen3.6-27B, Gemma4-31B, and proprietary Gemini3-Flash. The deployed monitor backbone is a 0.6B Qwen3-0.6B adapted via LoRA; some experiments train monitors specifically per policy and others train a single monitor on Qwen3.6-27B trajectories and transfer it to other policies.
Themesproductivity human_ai_collab
IdentificationControlled benchmark experiments on SWE-bench Verified: train/validation/test splits, out-of-fold monitor training with Platt calibration and operating-point selection to target false-positive rates; comparative baselines (AgentStop, Duration, Cold restart, SWE-PRM) and ablations; transfer tests of a monitor trained on one policy evaluated on other policies. No causal identification strategy for real-world economic outcomes beyond within-benchmark counterfactual-style comparisons.
GeneralizabilityBenchmark-to-real-world gap: results are on SWE-bench Verified and may not reflect production repositories, diverse engineering workflows, CI environments, or human review patterns., Model family limitations: evaluated on a small set of LLM families and scales; behavior may differ for other architectures or substantially different model scales., Task domain narrowness: focuses on repository-level software-engineering agent tasks; conclusions may not generalize to other agentic domains or to tasks that create many new files (new-file edits are excluded from overlays)., Overlay assumptions: the restart design excludes newly created files and assumes git-apply compatible edits; in many real projects edits may be more complex or require semantic context., Economic extrapolation: compute/energy savings are estimated from FLOP-per-token scaling rather than measured power consumption in deployed systems, limiting direct economic or carbon-impact claims.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| FailFast saves 14.6%–20.4% of execution tokens at a target 5% false-positive rate across four SWE-agent policy models. Organizational Efficiency | positive | Fraction of total agent execution tokens reclaimed through early termination |
Reading fidelity
high
Study strength
medium
|
n=500
14.6%–20.4% token savings
|
| On Qwen3.6-27B, FailFast saves 20.4% of token compute at a 5% false-positive-rate budget, outperforming AgentStop's 12.5% and the Duration control's 11.4%. Organizational Efficiency | positive | Token compute reclaimed by early stopping |
Reading fidelity
high
Study strength
medium
|
n=500
20.4% versus 12.5% and 11.4%
|
| A monitor trained only on Qwen3.6-27B trajectories transfers without weight updates to three other policy models, including the closed-API Gemini 3 Flash model. Task Allocation | positive | Cross-model transfer of early failure prediction |
Reading fidelity
high
Study strength
medium
|
n=500
|
| At a 25% false-positive-rate target on Qwen3.6-27B, FailFast intercepts 68.3% of failing runs while reclaiming 49.0% of total tokens. Organizational Efficiency | positive | Failure detection recall and total token compute reclaimed |
Reading fidelity
high
Study strength
medium
|
n=500
68.3% failing-run interception; 49.0% token savings
|
| RestartSmart increases Qwen3.6-27B task resolution from 66.6% to 71.8% at a 25% false-positive-rate target. Task Completion Time | positive | SWE-bench task resolution rate |
Reading fidelity
high
Study strength
medium
|
n=500
66.6% to 71.8% (+5.2 percentage points)
|
| RestartSmart outperforms a cold restart under the same alarms: 71.8% resolution versus 66.8% for cold restart at a 25% false-positive-rate target. Task Allocation | positive | SWE-bench task resolution rate after restart |
Reading fidelity
high
Study strength
medium
|
n=500
71.8% versus 66.8% (+5.0 percentage points)
|
| At a 10% false-positive-rate target, RestartSmart improves resolution by 3.0–4.0 percentage points across three open-weight policies, with 20.5%–36.9% net token overhead. Organizational Efficiency | mixed | Task resolution rate and net token overhead from restarting |
Reading fidelity
high
Study strength
medium
|
n=500
3.0–4.0 percentage-point resolution gain; 20.5%–36.9% net token overhead
|
| Including monitor overhead, FailFast yields estimated inference-energy and operational-carbon savings of 14.5%–20.3% across the three open-weight policies. Organizational Efficiency | positive | Estimated inference energy and operational carbon emissions |
Reading fidelity
high
Study strength
low
|
n=500
14.5%–20.3% savings
|
| The F2P progress signal improves the monitor consistently, whereas the P2P progress signal provides no significant gain and is excluded from the deployed configuration. Decision Quality | mixed | Early failure-prediction performance |
Reading fidelity
high
Study strength
medium
|
not reported
|