GDPevo builds a large, automated benchmark of enterprise workflows that forces agents to infer and recombine hidden business rules; self-evolution improves held-out task accuracy by up to 16.44 percentage points, but evolved agents still fall far short of a 91.6% oracle ceiling.

Main Finding

GDPevo is a new, evolution-native benchmark and automated pipeline for measuring agent self-evolution on economically valuable, GDP-related enterprise workflows (CRM, ERP, finance, healthcare, legal, data). Its rule-hybridization design makes train→test gains attributable to learned business rules. Using GDPevo (240 tasks, 24 groups), the authors find that non‑parametric self-evolution (skill-based) consistently improves held-out test accuracy (up to +16.44 percentage points) and can reduce resource cost, but even the best evolved agents remain well below an oracle ceiling (91.6%), indicating substantial headroom for improvement. The pipeline, benchmark, and evaluations are publicly released.

Key Points

• Benchmark scope and scale
  • V1: 120 tasks (12 groups, 5 training + 5 test per group). V2: additional 120 tasks (total 240 tasks, 24 groups).
  • Domains: CRM, ERP, finance, healthcare, legal, and data-centric workflows—tasks chosen for economic relevance and rule-based evaluability.
• Rule hybridization (core innovation)
  • Decompose each workflow into atomic, checkable business rules.
  • Scatter subsets of rules across 5 training tasks; recombine them across 5 held-out test tasks.
  • Ensures test-time improvement requires inferring and composing rules learned during training—makes transfer attributable.
• Fully automated construction pipeline
  • Three stages: (1) seed scenario discovery from real-work benchmarks (GDPval, SOP-Bench, JobBench, etc.), (2) task-group generation (env + 5 train + 5 test), (3) calibration & independent review.
  • Calibration criteria: base (no training) test accuracy ≈ 40–60%, fewshot lift ≈ 0.1–0.3, final fewshot accuracy < 0.8.
  • Rapid regeneration allows practical defense against data contamination (V2 produced in under two days).
• Deterministic evaluation and metrics
  • Every task graded by a deterministic, rule-based grader (LLM used to convert rubric → code tests), enabling reproducible error attribution.
  • Cost treated as first-class metric: tokens, agent turns, and monetary cost reported alongside accuracy.
  • Diagnostic outputs: per-group breakdowns, radar views, transfer heatmaps.
• Evaluations and findings
  • Setup: four agents (harness + model) × four supervision types:
    • no‑evolution (base)
    • fewshot (training Q&A gold — akin to supervised fine-tuning)
    • reflect (training questions + scores of the agent’s own attempts — akin to RL with self-feedback)
    • self (training questions only — unsupervised)
  • Evolution method tested: skill-based (non‑parametric persistent skills); GDPevo is agnostic to evolution modality.
  • Main empirical results:
    • Self-evolution yields consistent held-out accuracy gains (max +16.44 pp).
    • Fewshot is the most reliable supervision for boosting in-domain accuracy but tends to overfit and can hurt cross-domain transfer.
    • Reflect transfers more robustly across groups (less overfitting than fewshot).
    • Minimal evolution machinery inside the harness already produces gains; much of the gain magnitude depends on the base model’s capabilities.
    • Despite improvements, best evolved agents are substantially below the oracle ceiling (91.6%), indicating current agents’ self-evolution is far from fully effective.

Data & Methods

• Data construction
  • Seed scenarios drawn from existing real-work benchmarks; agentic pipeline expands them into concrete examples and environments.
  • For each task group: shared environment (APIs, web app, DB), 5 training tasks exposing different rule subsets, 5 held-out test tasks that recombine rules.
  • Atomic rules are enterprise-specific and intentionally absent from general world knowledge.
• Automation & quality control
  • Orchestrator creates blueprints and a task–rule matrix; task builders generate individual tasks.
  • Calibration agent(s) ensure difficulty and learnability bands; 6 independent reviewer agents verify completeness, rubric quality, and absence of answer leakage (accept if ≥5/6).
• Evaluated configurations
  • Full experiments used combined V1+V2 (240 tasks).
  • Evolution method in experiments: skill creator + skill library carried into test tasks.
  • Deterministic graders implemented as executable tests derived from rubric elements.
• Metrics recorded
  • Accuracy per task and per group, token consumption, agent turns, monetary cost, transfer heatmaps, and per-group diagnostic breakdowns.
• Release
  • Pipeline, benchmark datasets (V1, V2), and full evaluation results published: https://github.com/Prism-Shadow/GDPevo

Implications for AI Economics

• Economic relevance of evaluation
  • GDPevo targets enterprise workflows with clear GDP impact (billing, invoicing, compliance, claims, record reconciliation), making measured improvements economically meaningful.
  • By reporting both accuracy and resource costs (tokens, turns, money), GDPevo enables assessment of the ROI of self-evolution strategies—not just technical performance.
• Design implications for firms deploying agents
  • Prefer evaluation frameworks that measure transfer (not just in-domain fine-tuning gains). Fewshot-like supervised evolution can boost in-domain performance but risks overfitting and poor cross-task generalization.
  • Reflective/self-feedback mechanisms may produce more robust cross-domain transfer, which is important for heterogeneous enterprise workloads.
  • Non‑parametric persistence (skills, memories, prompts) can yield measurable gains without full model fine-tuning—potentially lower-risk for deployment—but current gains leave substantial headroom.
• Benchmarking and contamination risk
  • Automated regeneration is essential: economic benchmarks used in production must be refreshable to avoid data contamination and preserve validity for procurement/regulation decisions.
• Policy and investment signals
  • The sizable gap to the oracle ceiling (91.6%) indicates further R&D value: investments in better evolution methods (skill discovery, RL-style learning, continual fine-tuning) and stronger base models are likely to produce economically valuable improvements.
  • Firms should evaluate both accuracy uplift and operational cost reduction when deciding to adopt evolving agent architectures.
• Research & procurement recommendations
  • Procurement of agentic systems should require transfer-tested benchmarks (like GDPevo) rather than narrow supervised gains.
  • Cost metrics should be contractually tied to expected efficiency gains (token/turn/cost savings) in addition to success rates.
  • Use transfer heatmaps and per-domain breakdowns to identify where evolution adds value vs. where it might harm performance (to inform hybrid human+agent workflows).

If you want, I can (a) extract example task(s) that illustrate rule hybridization concretely, (b) summarize the empirical tables/plots (per-domain gains, transfer matrices), or (c) draft short guidance for enterprise procurement teams based on these results.