Backbone LLM choice, not agent wiring, largely determines how well general-purpose agents perform; within-model architecture swaps still move scores by up to 12 percentage points. Open-weight models repeatedly collapse on certain agent architectures or benchmarks, exposing failure modes hidden by aggregate metrics.

Main Finding

The paper introduces a Unified Protocol and Exgentic evaluation harness to run unmodified general-purpose agents across heterogeneous benchmarks, and uses them to produce the first Open General Agent Leaderboard (5 agent architectures × 5 backbone LLMs × 6 benchmarks). Key empirical findings: (1) general-purpose agents adapt across diverse domains without per-domain engineering; (2) backbone model choice dominates aggregate performance variance (27.8% vs 0.5% for architecture), but architecture can cause large within-model swings (up to ~12 percentage points); (3) on 4/6 benchmarks top general agents match heavily-customized specialist agents; (4) open-weight backbones exhibit brittle “generality sinks” (architecture- or benchmark-specific collapses) absent from frontier closed-source models; (5) architectures produce distinctive failure signatures that aggregate success rates hide. Total evaluation cost reported ≈ $20K.

Key Points

• Unified Protocol: A minimal mediation format (task / context / actions) that represents agent–benchmark interactions across CLI, tool-calling, MCP, code-generation, and conversational protocols. Allows plug-and-play of agents and benchmarks without per-case rewiring.
• Exgentic harness: Implements the Unified Protocol and runs the original agent implementations and benchmarks as black boxes in isolated sessions.
• Scope of evaluation:
  • Agents: ReAct, ReAct Short (tool shortlisting), Smolagent (code-generation), OpenAI Solo (MCP), Claude Code.
  • Backbone LLMs: three closed-source frontier models (Claude Opus 4.5, Gemini 3 Pro, GPT 5.2) and two open-weight models (DeepSeek‑V3.2, Kimi‑K2.5).
  • Benchmarks: BrowseComp+, τ2-Bench (Retail/Airline/Telecom), SWE-Bench Verified (mini-swe), AppWorld. Tasks: ~100 per benchmark (50 for one subdomain); 100-turn cap per task.
  • Full factorial: 5 architectures × 5 models × 6 benchmarks → 150 configurations; ~550 tasks aggregated per point in cost-performance plots.
• Leaderboard results and costs:
  • Top-performing configs are closed-source model pairings (Claude Opus 4.5 dominates). Example: OpenAI Solo + Claude Opus 4.5 averaged ~0.73 success rate at ≈ $8.5/task.
  • Cost-efficient end occupied by some GPT 5.2 and open-weight-backed configs with much lower per-task cost (sub-dollar to low-dollar range).
  • Total evaluation spend ≈ $20K.
• Performance patterns:
  • Model choice explains most variance (27.8%); architecture explains little in aggregate (0.5%), yet within a single backbone model architecture changes can swing results by up to ~12pp (and larger swings for open models).
  • On 4/6 benchmarks, best generalists are statistically indistinguishable from top published specialists; BrowseComp+ and τ2-Bench-Telecom remain specialist-led.
  • Open-weight models showed sharp failures: some architectures or benchmarks caused near-zero performance (termed “generality sinks”), traceable to protocol violations or API limits (e.g., tool limits).
• Failure-mode structure:
  • Different architectures fail in characteristic ways (e.g., Claude Code / OpenAI Solo tend to stop early; ReAct variants may skip evidence gathering).
  • Aggregate success rates miss these qualitative differences; the paper argues behavioral trace analysis is necessary.

Data & Methods

• Unified Protocol: Instances with three fields — task (instructions), context (documents/policy/state), actions (typed action set with parameters and observations). Optional message and final-answer action flags support common UI flows.
• Benchmarks adapted without changing original semantics; wrappers expose native actions via the Unified Protocol.
• Benchmarks:
  • BrowseComp+ (research information search; fixed retriever to isolate agent reasoning).
  • τ2-Bench (simulated customer service across retail/airline/telecom; user utterances + tool actions).
  • SWE-Bench Verified (human-validated software engineering tasks; sandboxed bash action / git patches).
  • AppWorld (realistic user-assistance; Python interpreter + APIs).
• Agents: used off-the-shelf agent bundles (scaffold, tools, memory, schema guards, etc.) exactly as shipped by authors; ReAct Short includes a tool shortlisting component treated as an architectural variant.
• Backbone models: three closed-source frontier models and two open-weight models; default provider sampling settings used (no hyperparameter tuning to avoid confounding).
• Metrics:
  • Success rate (benchmark-defined pass/fail), per-task cost, average steps, bench-weighted mean (balanced across benchmarks), cost-efficiency (bench-weighted success / $ per task).
• Statistical analysis:
  • Variance decomposition using η²; paired t-tests for model/architecture comparisons; McNemar pooled tests for leaderboard claims; multiplicity corrections (Benjamini–Hochberg/Yekutieli). Confidence intervals and procedures detailed in appendices.

Implications for AI Economics

• Procurement and cost-performance tradeoffs:
  • The study quantifies cost vs performance across model+agent combos: high-performing closed-source backbones (e.g., Claude Opus 4.5) deliver best absolute success but at much higher per-task cost; some frontier and open models offer more cost-efficient points. Buyers should evaluate cost-efficiency (success per dollar) not just raw accuracy.
  • The Pareto frontier structure implies distinct market segments: high-cost/high-performance offerings versus low-cost/cost-efficient alternatives. This shapes procurement decisions in enterprise deployments and pricing strategies of model providers.
• Vendor dominance and concentration risks:
  • Model choice accounts for most variance; top closed-source backbones lead across architectures. This suggests strategic dependence on a small set of top providers, with potential vendor lock-in and market concentration. Policymakers and firms should consider competition and interoperability.
• Value of architecture and tooling:
  • Although architecture explains little aggregated variance, it can be decisive within a given backbone—so investment in agent architectures (tool shortlisting, memory, schema guards) matters, especially to mitigate limitations of specific models or APIs. There is potential market value for middleware (tool-selection, robust protocol adapters) that improves robustness across backbones.
• Open-weight models and fragility:
  • The observed “generality sinks” in open models highlight deployment risk: open-weight models may be economical but brittle for some agent patterns or heavy-action environments. Economic decisions (e.g., choosing open models to save costs) need to weigh risk/variance and the cost of fallback engineering.
• Labor-substitution and productization:
  • Finding that generalists match specialists on most benchmarks implies lower marginal cost to deploy agents across domains (reduced need for expensive per-domain customization). This could accelerate automation and substitute for specialized software solutions or human labor in many tasks, lowering costs but also concentrating rents with model providers and integrators.
• Standardization lowers integration friction (lower transaction costs):
  • The Unified Protocol and Exgentic harness reduce the integration cost of evaluating and deploying agents across tasks. Reduced transaction and integration costs can accelerate competition and innovation in agent tooling, and allow buyers to compare providers more objectively.
• Evaluation and market signaling:
  • Public leaderboards and trace-level analyses provide richer market signals than single-number benchmarks. For economic decision-making, traceable failure modes (stability, protocol violations, tool limits) matter because they affect total cost of ownership (engineering time, reliability, compliance risk).
• Regulatory and policy considerations:
  • Concentration around a few high-performing closed-source models plus brittleness in open models argues for transparency and interoperability standards to reduce systemic risk and enable contestability. Regulators may need to monitor dependency risks for critical services automated by general agents.
• Recommended metrics for procurement and research economics:
  • Use cost-efficiency (success per dollar), failure-mode diversity, and robustness across action-space sizes as procurement criteria. Marginal engineering costs to mitigate model-specific failures should be quantified when comparing open vs closed solutions.

Short takeaway: evaluating agents across heterogeneous, realistic workloads reveals that backbone model choice largely drives value, agent architecture can materially change outcomes within a given model, and cost-efficiency tradeoffs plus robustness differences (closed vs open) are central for economic decisions about deploying general-purpose agents.