A new benchmark reveals LLM agents are less production-ready than single-run scores imply: semantically equivalent input changes and simulated API faults cut success rates substantially, with rate limiting the most damaging failure mode; ReAct agents and Gemini 2.0 Flash prove more robust and cost-efficient than competitors.

Main Finding

ReliabilityBench introduces a practical, production-focused evaluation framework for tool-using LLM agents that measures reliability across three interacting dimensions—consistency (k), robustness to input perturbations (ε), and fault tolerance to infrastructure failures (λ)—and shows that common single-run metrics (pass@1) materially overestimate production reliability. Key empirical findings: paraphrase/distractor perturbations reduce success rates by ~8.8% (96.9% → 88.1%), rate-limit faults cause the largest single-mode degradation, simpler ReAct agents outperform more complex Reflexion agents under stress, and a much cheaper model (Gemini 2.0 Flash) delivered comparable reliability to GPT-4o at ~1/82 the cost.

Key Points

• Reliability dimensions:
  • Consistency (k): repeated-run success (passk).
  • Robustness (ε): intensity of task-description perturbations (synonyms, reordering, distractors, paraphrase).
  • Fault tolerance (λ): probability and types of tool/API faults (timeouts, rate limits, partial responses, schema drift).
• Unified metric: R(k, ε, λ) — the 3D reliability surface; derived metrics include surface volume, degradation gradients, and critical thresholds.
• Action Metamorphic Relations: correctness judged by end-state equivalence (state-based oracles) rather than textual similarity; this better captures semantic task correctness for agents.
• Chaos-style fault injection: configurable, realistic fault types (recoverable and non-recoverable) injected per tool call; fault profiles at light/medium/heavy intensities.
• Empirical findings (1,280 episodes total; main experiments: 480 episodes per model):
  • Perturbations (ε = 0 → 0.2) caused ~8.8% drop in pass rate.
  • Rate-limit-only ablation produced the largest negative impact (−2.5% vs mixed baseline).
  • ReAct had higher surface volume and better recovery stats than Reflexion (fewer extra tool calls, higher successful recoveries).
  • Cost comparison: Gemini 2.0 Flash ≈ $0.12 for 480 episodes (≈$0.00025/episode) vs GPT-4o ≈ $9.77 for 480 episodes (≈$0.0204/episode) — ~82× cheaper for similar reliability (~−0.6% difference).
• Limitations noted by authors: scale (1,280 episodes), limited model set, simulated faults and domains.

Data & Methods

• Domains: Scheduling, Travel, Customer Support, E-commerce. Each domain uses a state dictionary and deterministic state-based verifiers (no LLM judging).
• Tools: 25+ domain-specific tool functions (e.g., book meeting, search flights, create order).
• Agents/architectures: ReAct and Reflexion, both using structured function calling (JSON).
• Models: Gemini 2.0 Flash (primary) and GPT-4o (comparison).
• Experimental grid:
  • Trials per config: k = 2 (pass2 metric reported).
  • Perturbation levels ε ∈ {0.0, 0.1, 0.2} (light to medium).
  • Fault intensity λ ∈ {0.0, 0.2} (baseline and medium).
  • Fault types for ablation: timeout, rate-limit, partial-response, mixed.
• Episodes:
  • Main: 480 per model (20 tasks × 3 ε × 2 λ × 2 agents × 2 runs).
  • Ablations: 320 episodes.
  • Grand total: 1,280 episodes.
• Metrics:
  • passk (probability all k runs succeed).
  • R(k, ε, λ) reliability surface; surface volume and gradients used to compare designs.
  • Operational metrics: additional tool calls after faults, successful recoveries, per-episode token counts & costs.
• Cost measurement: December 2024 price points used; reported token counts (~1.19M tokens per model across 480 episodes).

Implications for AI Economics

• Cost-per-success and model selection:
  • Use expected-cost-per-success = cost_per_run / P(success) to compare deployment options rather than per-run price alone. Example (approximate from paper): Gemini ~ $0.00025/run and ~96.9% success → ~$0.00026 expected cost per successful task; GPT-4o ~ $0.0204/run and ~95% success → ~$0.0215 expected cost per successful task. That yields ~80–85× cost advantage per successful task for the cheaper model in these experiments.
  • When models have similar reliability, cheaper models dramatically lower unit economics at scale.
• Architecture complexity vs operational cost:
  • More complex reasoning architectures (Reflexion) can add failure modes and increase retry/tool-call counts, raising token usage and operational cost; simpler architectures (ReAct) may be more cost-effective for reliability-sensitive production workloads.
  • Extra reflection/retry logic can increase mean tokens per episode (and latency) and may not improve, and can even worsen, fault-robustness.
• Hidden costs from over-optimistic benchmarks:
  • Single-run pass@1 overestimates production reliability (authors report 20–40% overestimation elsewhere). Economic planning, SLAs, and capacity planning that rely on pass@1 will underbudget for retries, human intervention, error handling, and customer-impact costs.
• Faults, retries, and SLA design:
  • Rate limits and API-specific faults impose asymmetric costs and failure modes—agents that abandon on rate limits impose direct failure costs; agents that retry increase token usage and latency (cost and user experience trade-offs).
  • Use reliability surfaces to set SLAs: choose k (redundancy/retry budget) and acceptable ε, λ scenarios to meet target reliability and compute marginal cost of increased k or improved tooling.
• Procurement and pricing signals:
  • Benchmarking with R(k, ε, λ) provides purchasers (enterprises) a clearer basis to negotiate pricing: vendors should be evaluated on robust reliability under perturbations and faults, not only on peak accuracy.
  • There is an economic incentive for model providers to optimize for robust, low-cost inference for tool-using agent workloads (token efficiency, deterministic tool integration patterns).
• Risk management and capital allocation:
  • Firms should allocate resources to (a) agent-level reliability testing (ReliabilityBench-style), (b) operational mitigations (robust retry/backoff, caching, circuit breakers), and (c) monitoring to detect schema drift and emergent failure modes—these reduce downstream user-costs and liability.
• Practical suggestions for economic decision-making:
  • Measure cost per successful task under realistic ε and λ scenarios before selecting model/architecture.
  • Evaluate whether added model cost (higher-tier LLM) yields marginal reliability gains that justify the price — often it may not.
  • Invest in engineering (retry/backoff, schema validation, tool contracts) where it reduces failure-caused revenue loss more than switching to a higher-cost model.
  • Use R(k, ε, λ) (or surface volume) as a procurement KPI to compare vendor/model/agent-stack trade-offs.
• Broader market effects:
  • If benchmarks like ReliabilityBench become standard, buyers may shift demand to models and vendors that demonstrate robust, low-cost reliability on practical agent tasks—this could reshape pricing competitiveness away from maximum-capability, high-cost offerings toward efficient, robust models optimized for tool use.

Limitations to bear in mind when applying these economic conclusions: the experiments cover limited domains, a small sample of models, and simulated faults; cost numbers depend on pricing snapshots (Dec 2024). Still, the methodological point stands: measuring multi-dimensional reliability under realistic stress is essential for accurate cost/risk estimation when deploying LLM agents at scale.