An evolutionary meta-agent composes a menu of LLMs to meet price–accuracy targets and captures almost the entire Pareto frontier on two public benchmarks, frequently outperforming hand-built systems while using only tens of labeled examples and modest compute.

Main Finding

RoboPhD—an evolutionary meta-agent that evolves complete Python agent programs over a priced menu of LLM endpoints—can discover model compositions (cascades, ensembles, stage-wise model choice) that Pareto-dominate almost all public competitors on both quality and inference-cost axes. Using training pools of only 66–100 examples and modest compute ($34–$368 per evolutionary run), RoboPhD produced agents that occupy nearly every slot of the AstaBench leaderboards’ Pareto frontiers for two semantically distinct tasks (DS-1000 code generation and PaperFindingBench document retrieval).

Key Points

• Goal: produce agents that maximize accuracy subject to an operator-chosen average per-problem cost target τ (so a vendor can aim to beat competitors at every price point).
• Cost-aware objective: training score S combines mean quality and a graded cost penalty: S = (100 / n) [ sum_i q_i − max(0, (c̄ − τ) / κ ) ], where q_i are per-example quality scores, c̄ is batch mean cost, τ the target, and κ a slope (default κ = 0.1τ). The cost penalty is only applied during training to steer evolution; leaderboard evaluations report raw score and cost separately.
• Evolution mechanism: single-file agent programs are iteratively evolved in an LLM-based coding environment (Claude Code) with file/shell access; candidates are evaluated on fresh random batches and ranked by Elo in head-to-head tournaments (avoids a fixed validation set).
• Menu-driven composition: agents use a registry of nine priced LLM endpoints (three each from OpenAI, Anthropic, Google) with a per-call reasoning_effort knob. Evolution discovered compositions using between 1 and 5 models per agent, exploiting cheap models for routine work and escalating to stronger models for hard cases.
• Minimal human engineering: inputs were limited to (i) small objective document, (ii) a short background doc (130–227 lines) describing task, scoring, APIs, and (iii) a tiny seed agent (46–121 lines). Evolved agents ranged ~302–2,781 lines.
• Data-starved regime: training pools were small (DS-1000: 100 examples; PaperFindingBench: 66 examples), with batches of 14–20 and total data reuse ≈4× the pool. The objective explicitly instructed the optimizer to generalize (to mitigate memorization risk).
• Results (official Ai2/AstaBench scoring snapshot 2026-08-16):
  • DS-1000 (900 held-out): RoboPhD held 6 of 7 Pareto frontier points; top RoboPhD entry 0.862 accuracy @ $0.127/problem. Several RoboPhD entries dominated other general-purpose and hand-engineered systems; the only non-RoboPhD frontier survivor was a hand-built system ("Button").
  • PaperFindingBench (267 held-out): RoboPhD held 5 of 6 Pareto frontier points, including the board leader 0.440 @ $0.279 and the cheapest frontier point 0.220 @ $0.006. RoboPhD also Pareto-dominated some hand-built systems (including Asta v0) and all general-purpose scaffold entries.
• Examples of discovered designs:
  • Cheap-first cascades with strong-model escalation and adjudication.
  • Mixed ensembles that compare outputs from different model families and escalate only on disagreement.
  • Stage-wise model choice (different models per pipeline stage: planning, retrieval, grading).
• Cost of running an evolutionary campaign is modest (per-run compute $34–$368); human authoring work is low compared to hand-engineered baselines (e.g., a 14-person team credited for Asta Paper Finder).

Data & Methods

• Tasks:
  • DS-1000: 1,000 data-science code completion problems (Stack Overflow-derived, execution-checked tests), binary scoring, 100 training / 900 held-out.
  • PaperFindingBench: 267 literature-search queries (three query types: specific, metadata, semantic); scoring is adjusted micro-F1: ~27% exact-match (ID intersection), ~73% LLM-judged over agent-supplied evidence; 66 training / 267 held-out.
• Tooling: Standard Asta MCP corpus API (8 retrieval tools) and a sandboxed Python executor (free). Only LLM usage is metered in cost calculations.
• Evolution setup:
  • Candidate single-file agent programs evaluated on batches (14–20 examples), with fresh random batches drawn each iteration.
  • Selection via Elo over head-to-head batch tournaments.
  • Evolution environment (Claude Code) allowed the evolution process to read incumbent code and per-example diagnostics and write the next candidate.
  • Diversity mechanisms and controlled comparisons to other evolution/research systems are reported in Borthwick et al. (2026) (not repeated in detail here).
• Cost-targeting:
  • Operator picks τ (often set at a competitor’s price). κ (cost_per_error) sets how many dollars of overage equal one error-equivalent; default κ = 0.1τ used in reported runs.
  • Penalty shape is linear above τ (graded), flat below τ—to avoid catastrophic cliffs from unlucky batches and to let evolution operate near the boundary.
• Data-starvation countermeasures:
  • Explicit instruction to the optimizer to treat visible batch as training signal, not target.
  • Fresh-batch sampling + Elo ranking to reduce overfitting incentives.
  • Empirical monitoring at multiple waypoints showed monotonic improvements on held-out score alongside training selection (evidence against catastrophic overfitting in reported runs).
• Compute and artifacts:
  • Evolved agents length: ~302–2,781 lines.
  • Menu of nine LLM endpoints spanned ~10× per-token price; agents used between 1–5 handles.
  • Per-evolution-run compute cost: $34–$368.

Implications for AI Economics

• Market competition and product design:
  • Vendors can algorithmically produce offerings that dominate competitors across both price and quality axes by evolving agents that compose multiple models and adapt effort per call. This raises the bar for incumbents: simply offering a single-model product at various prices may be insufficient.
  • The ability to hit specific price-quality operating points cheaply (low human engineering, modest compute) lowers the marginal cost of entering niches and competing at multiple price points, increasing competitive pressure and likely compressing margins for specialized agent offerings.
• Provider pricing & strategy:
  • Menu-based composition amplifies the economic value of cheaper model tiers: even very cheap models can be leveraged for routine work and adjudication, making price tiers strategic complements rather than substitutes. Providers may respond by adjusting price schedules, per-call pricing, or rate-limiting multi-model orchestration patterns.
  • LLM providers could react with product-design choices (e.g., restricting composition across providers, introducing bundled multi-model offerings, or changing per-call parameters and metering) to capture more value or limit arbitrage by evolved agents.
• Labor and development economics:
  • The low human cost to produce high-quality, cost-targeted agents could reduce demand for large expert engineering teams for many applied tasks, shifting labor toward higher-level oversight, tool maintenance, or proprietary data curation.
  • Conversely, firms that specialize in building and operating evolution platforms (or in curating high-quality task/evaluation docs) could capture new service value.
• Benchmarking and entry barriers:
  • Public leaderboards (like AstaBench) enable automated systems to optimize directly against market-visible cost/score axes; this reduces information asymmetries and accelerates competitive iteration. Maintaining robust, multi-axis benchmarking becomes an economic policy lever.
• Risks and regulatory considerations:
  • Fast automated Pareto improvements could intensify arms races among agents and providers, raising systemic risk (exposure to model failures at scale, incentives to obscure pricing or capability details).
  • Concentration risk: if only a few platforms supply high-quality multi-model menus or restrict access, upstream control could recreate market power even if downstream agent-design is automated.
  • Potential negative externalities (misuse, displacement) grow when high-performance agents can be cheaply produced across many tasks; policymakers and firms should evaluate governance, access controls, and transparency.
• Strategic recommendations for firms:
  • Firms selling agentic products should consider exposing flexible priced-model menus, metering primitives, and tooling to enable composition while aligning monetization (e.g., custom bundles, composability-aware pricing).
  • Buyers should benchmark offerings on both accuracy and operational cost (average per-query), not just top-line accuracy.
  • Firms should invest in proprietary data, tooling integration, or access restrictions if they wish to sustain a competitive moat against automated evolutionary optimization.

Limitations and caveats - Results are demonstrated on two AstaBench tasks; generalization to all real-world tasks is suggestive but not proven. - The approach depends on availability of priced model endpoints and a menu spanning cost/quality; closed ecosystems or changed pricing may alter viability. - Evolutionary optimization can exploit quirks in APIs or evaluators; robust production deployment will require monitoring, maintenance, and possibly human-in-the-loop checks.

Bottom line: automated, cost-targeted evolution over multi-provider LLM menus can cheaply discover sophisticated, cost-effective agent architectures that dominate many existing offerings on public leaderboards—this has important competitive and pricing implications for vendors, platform providers, and buyers in the AI agent economy.