A new resource-aware benchmark exposes a production gap: domain-specialized agents achieve high success while meeting strict latency, cost and memory budgets, whereas popular general-purpose agent methods frequently fail or run over budget.

Main Finding

AGENTSLABENCH is a resource-aware, reproducible benchmarking framework that evaluates autonomous LLM agents not just on task correctness but jointly on latency, cost, compute, memory, and network use under declared resource budgets. Using Docker-enforced budgets and a standardized profiling protocol, the authors show that task-specialized agents substantially outperform general-purpose baselines on production-relevant, resource-constrained criteria: specialized agents hit high success rates while meeting budgets; general baselines often fail entirely on domain tasks. They introduce the Efficiency-Adjusted Success Rate (EASR) — success weighted by adherence to declared budgets — to distinguish “success at any cost” from production-viable success.

Key Points

• Resource-first evaluation: Each task declares CPU, memory, wall-time, and network budgets enforced by Docker; over-budget episodes are terminated and recorded as failures.
• Multi-dimensional profile per episode: correctness (success, exact_match, f1), latency (wall-time, p50/p95), cost (USD, tokens), compute (peak RSS, CPU time), network (HTTP calls, bytes), safety violations. Output: per-episode JSONL.
• EASR metric: EASR = success × min(1, budget_lat / actual_lat) × min(1, budget_cost / actual_cost) × min(1, budget_mem / actual_mem). Rewards success only when within declared budgets.
• Tasks and scale: v1.0 contains 16 tasks across 6 categories (5 core tasks: multi-hop QA, retail substitution, code generation, web shopping, travel planning), ~3,500 test samples, sealed test sets (SHA256), multi-seed runs.
• Agents evaluated: 5 general baselines (ReAct, PlanAndSolve, Reflexion, Chain-of-Thought, Random) and 4 task-specialized agents (RetailAgent, WebShoppingAgent, TravelPlanAgent, CodeGenAgent).
• Main empirical results:
  • Specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning reported as 100% for web_shop/fact_qa; travel and some others 83.3–66.7%), and meet budgets.
  • General baselines have 0% success on 4/5 domain tasks (they may do well only on fact_qa).
  • Example contrast (paper): RetailAgent — 83.3% success at median 142 ms, $0.002, 180 MB; a ReAct+GPT‑4 solution might show 78% accuracy but median 3.2 s, $0.12, 1.2 GB (violating production SLAs).
• Failure modes: hallucinated tool calls, timeouts on complex multi-step tasks, constraint conflicts for planning tasks; detailed failure-mode distributions are provided.
• Reproducibility & infrastructure: Docker-isolated tasks, REST task interface (/reset, /step), sealed test sets, per-episode JSONL outputs, and released artifacts.

Data & Methods

• Task catalog: 16 distinct environments, 3,500 test samples total, core tasks include:
  • fact_qa_01 (multi-hop QA), retail_01 (product substitution), code_gen_01 (API-based code generation), web_shop_01 (find & purchase), travel_plan_01 (multi-constraint planning).
• Declared budgets: per-task CPU cores, memory, wall-time, network policy (allowed/disabled). Budgets enforced via Docker flags (-cpus, -memory, network policy, timeout).
• Profiling protocol (per episode): start container with limits → /reset → agent interacts via /step until done/timeout → measure container stats and API logs → judge correctness with LLM judge → compute EASR → write JSONL profile.
• Measured metrics: wall_time_ms, step latencies (p50/p95/p99), api_cost_usd, token counts, peak_rss_mb, cpu_time_s, http_calls, net_bytes, safety violations, trajectory, and various task-specific scores (e.g., price_optimality).
• Agents: 5 general-purpose prompting/architectural baselines (ReAct, PlanAndSolve, Reflexion, CoT, Random) and 4 domain-specialized agents implementing heuristics, caching, schema-aware tooling, and template logic.
• Experimental setup: 3 seeds per agent-task (default seeds 42, 123, 456), bootstrap 95% CIs reported (authors note wide CIs due to N=3 and recommend N≥10).
• Output & reproducibility: sealed test set hashes, Docker images, JSONL profiles, and instrumentation released for replication.

Implications for AI Economics

• Production viability and procurement decisions: EASR (or similar resource-aware metrics) should be incorporated into procurement, vendor selection, and SLA contracts. Buyers who select models solely on accuracy risk high operational costs (API charges, CPU/GPU time) and SLA violations; EASR provides a single interpretable signal for “deployable” performance.
• Pricing and monetization: Cost-per-query matters materially. The paper shows that a narrow, well‑engineered specialized agent can deliver similar or better task outcomes at a fraction of per-request monetary and latency cost compared to large LLM-based, unconstrained solutions. This supports business cases for investing in specialized agents or hybrid architectures (small local models + targeted tool calls) rather than paying for larger models per call.
• Investment and engineering trade-offs: The economic value of specialization (engineering effort to embed domain logic, caching, and schema-aware tool use) is quantifiable: it lowers operational expenditures (OPEX), improves latency (P99 SLA compliance), and reduces required infrastructure (smaller memory/CPU footprints). Firms must weigh upfront engineering (capex) versus ongoing API/model costs (opex).
• Market structure & product design: AGENTSLABENCH suggests a two-tier market: (1) commodity upstream LLM APIs used for general reasoning and rare tasks, and (2) productized, specialized agents optimized for unit economics and SLAs. Specialized agents can capture higher margins or enable products that large-model-only solutions cannot (due to latency/cost constraints).
• Resource allocation and capacity planning: Per-episode profiles enable forecasting of cluster costs and capacity needs, informing pricing models, throttling policies, and dynamic bidding for cloud resources. EASR-driven selection can minimize unit costs and aggregate compute/energy demand.
• Externalities and regulation: Resource-aware evaluation foregrounds compute and energy usage per task, aligning with environmental accounting (Green AI) and potential regulatory/contractual disclosure of per-transaction resource use or carbon footprint. Buyers and regulators may prefer agents that trade some accuracy for lower resource externalities.
• Platform & market signals: Benchmarks that publish multi-dimensional profiles (success × resource use) will shift market incentives away from “accuracy-only” leaderboards toward Pareto-efficient solutions. This can change price competition (e.g., cheaper, faster specialized agents winning real-world use cases) and encourage vendors to offer tiered products (low-latency low-cost vs. high-cost high-accuracy).
• Operational risk & contract design: Using EASR as a gating metric in CI/CD can reduce deployment risk and SLA penalties; contracts could include EASR guarantees and penalties for over-budget behavior (much like throughput/latency SLAs in existing cloud contracts).
• Limitations relevant to economics: Current budgets in AGENTSLABENCH are heuristic and CPU-only; real-world procurement should calibrate budgets to production PnS targets (P95/P99). Small-N seed uncertainty implies early-stage measurement error — economic decisions should account for statistical uncertainty and consider larger sample profiling.

Overall, AGENTSLABENCH provides an operationally and economically meaningful framework for choosing and optimizing agents under real-world constraints. For AI economics, the key takeaway is that accuracy alone is insufficient as a valuation metric — resource-aware success (EASR) better captures deployable value, cost-effectiveness, and the true economic trade-offs of agent design.