A new benchmark finds LLM agents routinely mishandle limited budgets: tool-API agents either overspend or stop short, achieving at most 24% strict budget success and under 7.3% economic-consistency, while workspace-configured systems do better but still expose a trade-off between completion and economical decision-making.

Main Finding

EcoAgent-Bench shows that task completion accuracy alone conceals economically irrational agent behavior under priced actions and explicit budgets. Many LLM agents either over‑escalate (wasting budget) or under‑escalate/abstain (failing to get necessary evidence). A new diagnostic—economic consistency (the minimum of accuracy on upgrade‑oriented and save‑oriented task groups)—reveals these one‑sided policies. Tool‑API agents perform very poorly on budgeted, cost‑sensitive decision making; workspace agents do better but still leave substantial room for improvement.

Key Points

• Benchmark novelty: every task includes a priced action set and an explicit budget; the preferred action depends on evidence and remaining budget (not a post hoc cost log).
• Task bundle: 304 real‑derived tasks across 5 families and 4 economic decision types:
  • escalation_qa (115) — escalate retrieval when local evidence insufficient
  • model_upgrade_qa (100) — route to higher model tier vs. cheap model
  • cheap_qa (72) — avoid unnecessary escalation (cheap evidence suffices)
  • stop_loss (10) — abstain when premises unsupported
  • frozen_information_qa (7) — GAIA anchors (model routing)
• Shared priced actions (abstract cost units): inspect_tool 4, search_tools 8, local_keyword_search 18, read_document 22, web_search_snapshot 70, deep_research_agent 260, page_human_oncall 520.
• Utility model: U = R − λC if correct within budget; −λC if incorrect within budget; −∞ if cost > budget. Benchmark emphasizes conditional routing/stop decisions that maximize expected utility, not just correctness.
• New metric: economic‑consistency = min(accuracy on upgrade‑oriented group, accuracy on save‑oriented group). This penalizes policies that only do well on one regime (e.g., “always escalate”).
• Key failure modes observed:
  • Under‑escalation / premature abstention: many tool‑API agents stop or abstain instead of invoking needed (expensive) tools.
  • Over‑spending: some agents (or scripted always‑escalate policies) achieve high micro accuracy but blow budgets on save‑oriented tasks.
  • Ungrounded guessing: prior‑knowledge answers without evidence lead to incorrect low‑cost answers.
• Budget sensitivity: sweeping GPT‑5.4 across budgets that cross the escalation threshold changed its escalation rate from 0% to only 3% — agents were largely insensitive to budget changes.

Data & Methods

• Sources and transformation:
  • Seeds drawn from HotpotQA, MuSiQue, GAIA and related corpora.
  • Each seed transformed into a frozen environment (agent view) and evaluator‑only oracle material, with canonical (budget‑compliant) and contrasting high‑regret traces.
  • Deterministic selection, anti‑leakage controls, provenance checks, and human realism audits (≈15% of tasks).
• Verification checks:
  • Escalation tasks: confirmed that cheap search cannot reach the authoritative source while deep research can.
  • Model‑tier labels: small models tested to fail/ succeed deterministically; frontier models tested to succeed where required (labels robust to repeated trials).
  • Stop‑loss tasks: independently validated as false‑premise/unanswerable; scorer demands evidence‑backed abstention.
  • Cost perturbation sanity check: random multiplicative noise (U(1/R, R)) preserves canonical vs. high‑regret ordering in nearly all draws for R=2,3.
• Evaluated agents and controls:
  • Scripted oracle controls (for construct validity): CheapFirst, CopilotFirst, RetryLoop, BudgetAwarePlanner (these can read gold answers at oracle steps to test policy choices vs. budget).
  • Tool‑API LLM agents (evidence‑only composite mode): Sonnet, GPT‑5.4, Gemini 2.5 Pro (multi‑turn tool calling up to 10 turns).
  • Workspace CLI agents: Claude Code (Opus 4.8, Haiku) and Codex (GPT‑5.5, GPT‑5.4‑mini) in materialized workspaces (execution proxy used as systems diagnostic).
  • Judging: automated judge (claude‑sonnet‑4‑6, temp 0) with human re‑labeling check (96 samples) showing 91.7% agreement (κ = 0.833).
• Scoring:
  • Strict budgeted success requires correctness, applicable evidence access, and no budget violation.
  • Reported aggregates: micro strict success, mean ledger cost, cost‑penalized score Sλ, and economic‑consistency (Econ).
• Reproducibility: full bundle, transformation pipeline, frozen evaluation environments, and integrity‑bound result artifacts released.

Empirical Results (high‑level numbers)

• Tool‑API agents: micro strict success very low (reported range ~3.9–24.0%); economic‑consistency at most ≈7.3%. Frequent behaviors: abstention on many tasks that actually require escalation; either under‑use or over‑use of expensive tools leading to low cost‑adjusted utility.
• Workspace CLI agents: substantially higher economic consistency (≈44.6–53.6% across configurations). Best performer: Claude Code (Opus 4.8) — ~65.1% micro success, with strong performance on save‑oriented items (≈96%) and moderate on upgrade items (≈54%).
• Scripted controls: validate benchmark structure — e.g., always‑escalate or always‑cheap policies succeed on some families but fail badly on the opposite families; BudgetAwarePlanner stays within budget and demonstrates the realizability of balanced policies.
• Budget sweep: GPT‑5.4’s escalation behavior barely changes across budgets designed to straddle escalation costs (0% → 3% escalation rate), indicating weak budget‑conditioned decision making.

Implications for AI Economics

• Evaluation design:
  • Benchmarks for deployed agents should make cost and budget explicit parts of task instances (not an afterthought) to measure economically rational behavior.
  • Micro accuracy alone is insufficient; metrics like economic‑consistency and cost‑penalized utilities are required to detect one‑sided, wasteful, or under‑escalating policies.
• Agent modeling and training:
  • Agents need mechanisms for value‑of‑information estimation, cost‑sensitive planning, and calibrated abstention to trade off correctness vs. expenditure.
  • Simple heuristics (always escalate / always use cheapest tool) are not acceptable in mixed regimes; learning or planning approaches should condition on both evidence quality and remaining budget.
  • Model‑tier routing (cascade/cost‑aware model selection) and stop‑loss (abstention) deserve more explicit objective design (e.g., reward shaping that penalizes unnecessary cost and rewards justified escalation).
• Deployment and economics:
  • Off‑the‑shelf LLM tool‑calling setups can either overspend for marginal gains or fail silently by under‑escalating; both have real economic consequences (cost blowouts, unmet user needs).
  • Providers and integrators should instrument agents to record action economics and include budget constraints in policy optimization and safety checks.
• Research directions:
  • Develop budget‑aware RL or planning methods that maximize expected utility as defined (correctness reward minus cost).
  • Better diagnostics and training data for stop/abstain behavior to avoid hallucination under insufficient evidence.
  • Study transfer of cost schedules (abstract units → real prices) and robustness of learned policies to price changes.
  • Use EcoAgent‑Bench (released assets) as a standard testbed for cost‑sensitive agent evaluation and for benchmarking model‑tier routing, tool orchestration, and abstention strategies.

Released artifacts: the 304‑task bundle, transformation protocol, frozen agent/evaluator fixtures, and bound result artifacts to enable reproducible study of budgeted, cost‑aware agent behavior.