A longitudinal financial benchmark finds that AI agents that retain prior experience meaningfully improve professional outputs: across 120 real-case tasks, self-evolving agents raise rubric scores by about 9–19 points and cut compliance issues by 0.12–0.44 per task, with larger gains for later related cases.

Main Finding

FinEvo-Bench is a longitudinal benchmark that measures whether and how much agents improve from retained experience when performing professional financial workflows. Across 120 real-case-grounded tasks (20 business scenes × 6 tasks), four self-evolving agent scaffolds (same Qwen3.7‑Max backbone) show consistent positive self-evolution: final rubric scores rise by 9.33–19.37 points and financial compliance issues drop by 0.12–0.44 issues per task versus paired non-evolving controls. Letta achieves the highest evolved absolute score (91.65) and fewest compliance issues (0.09/task); Codex achieves the largest paired self-evolution gain (+19.37). Gains concentrate later within-scene occurrences (ranks 4–6), and among tested experience carriers, skill-only evolution performed best.

Key Points

• Benchmark design
  • 120 tasks across 20 business scenes (6 related but substantively distinct cases per scene).
  • Tasks are multi-file (775 input files total; mean 6.46 files/task) and request open‑ended professional deliverables (reports, assessments, recommendations).
  • Scene-level 100-point rubrics (quality + compliance) authored and cross‑checked by domain experts.
• Evaluation protocol
  • Three independently shuffled, globally interleaved streams to require retrieval amid unrelated intervening tasks.
  • Paired state-reset (non-evolving) controls to isolate gains attributable to retained experience.
  • Scoring by an independent automated judge (Claude Code backed by Claude Opus 4.6); judge validated against a financial expert (ICC(A,1) = 0.95; mean absolute diff 1.6 points).
  • Agents receive rubric-based feedback summaries (not full rubrics) and may reflect/retain distilled experience.
• Agent scaffolds and persistent representations
  • Four scaffolds compared: Claude Code, Codex, Letta, GenericAgent (all using Qwen3.7‑Max as backbone).
  • Experience carriers varied: global/project memory, editable memory blocks, reusable Markdown files, distilled skills/playbooks.
• Main quantitative results (averaged across three runs)
  • Score increases (evolved vs non-evolving): +9.33 to +19.37 points.
  • Compliance issues reduced by 0.12–0.44 issues/task.
  • Final evolved scores: Letta 91.65, Codex 91.17, Claude Code 89.47, GenericAgent 83.34.
  • Largest paired gain: Codex +19.37; smallest paired gain: GenericAgent +9.33.
  • Late within-scene ranks (4–6) gain 6.10–8.70 points more than early ranks (1–3).
  • In Claude Code ablation: skill-only evolution outperformed memory-only and combined carriers (score 93.71; compliance 0.05 issues/task).
  • Rubric feedback outperforms reference-answer feedback by 3.95–7.93 points and reduces compliance issues.
• Cost trade-offs (agent-side tokens per task, units = 10^4 tokens)
  • Total token cost per task: Claude Code 76.50, Codex 68.75, Letta 50.43, GenericAgent 21.78.
  • GenericAgent is cheapest but shows lower absolute performance and smaller gains; Letta achieves top absolute quality but with moderate token costs (high execution tokens because its core memories are always injected).

Data & Methods

• Dataset construction
  • Each scene s defined from: institution-provided scene description Ds, validated professional procedure Ps, and an eligible pool Celigible_s of institution/public cases.
  • Experts select six substantively different cases per scene; remove/replace sensitive identifiers while preserving analytical relationships.
  • Scene specification Ss encodes objective, required inputs, operations/checks, expected deliverable, and compliance constraints.
• Rubrics and quality control
  • Scene-level 100-point rubrics evaluate correct use of inputs, required analysis, supported conclusions, and explicit financial compliance checks (fabrication, unsupported claims, role overreach, guarantees).
  • Two-domain-expert construction and cross-application to the six tasks; automated judge validated against human expert scoring (ICC = 0.95).
• Experimental protocol
  • Three global permutations of the 120 tasks (interleaved scenes).
  • Evolving runs: for each task, agent executes → receive rubric-based feedback from judge → reflect/consolidate → persist distilled experience. Reflection/store may be in skill, memory, or document form depending on scaffold.
  • Non-evolving runs: identical setup but agent state is reset before every task (paired with each evolving run).
  • Metrics: task quality (rubric 0–100), mean compliance issues per task, paired gain ∆Score and ∆Comp across K=3 runs and N=120 tasks, and agent-side token cost (execution + reflection).
• Additional diagnostics
  • Measured experience activation counts (how often stored experience entered execution context).
  • Ablations for experience carriers (memory-only, skill-only, combined) and feedback formats (rubric vs reference answer).
  • Cross‑scene diagnostic and scene-isolated execution experiments.

Implications for AI Economics

• Quantifiable productivity and risk-reduction gains from persistent learning
  • Self-evolving agents deliver substantial measurable improvements in task quality and compliance across related professional cases. For financial institutions, these gains translate into higher-quality reports and fewer compliance violations — both of which have direct monetary and reputational value.
  • The paired-control design gives an attributionable measure of how much value accrues from retained experience (useful for ROI calculations).
• Trade-offs: performance vs operational cost
  • Higher evolved performance often requires non-trivial reflection/execution cost (token consumption; compute). Decision-makers must weigh marginal quality/compliance benefits against compute and latency costs. GenericAgent demonstrates a low-cost, lower-performing option; Letta shows strong absolute performance with moderate cost; Claude/Codex incur higher reflection costs.
  • Practical procurement/ROI analysis should compare expected reductions in costly compliance incidents and analyst hours to incremental compute and integration costs.
• Learning modality matters for economic value
  • Skill-only evolution outperformed memory-only or combined carriers in this setting, suggesting investing in mechanisms that distill reusable procedures or structured skills (playbooks, templates, checklists) can yield higher returns than raw episodic memory storage.
  • Rubric-style feedback (structured, criteria-based) produced larger gains than reference-answer feedback. Organizations should prefer structured, evaluative feedback pipelines (possibly human-curated rubrics) to accelerate agent learning.
• Deployment and governance considerations
  • The benchmark’s interleaved-stream setup mimics realistic operational settings with mixed unrelated tasks; agents that can selectively retrieve relevant experience amid noise are more valuable for real workflows.
  • Auditability and validation: high agreement between automated judge and human expert (ICC = 0.95) is encouraging for scalable evaluation, but institutions will still need transparency and human oversight, especially on compliance-sensitive decisions.
  • Regulatory risk: automatic adaptation raises governance questions (traceability of what was learned, risk of inappropriate generalization). The benchmark’s design—limiting agents’ access to full rubrics and requiring generalization across distinct cases—illustrates one way to encourage generalized improvements rather than memorization of specific answers.
• Labor-market and organizational effects
  • Self-evolving agents could augment financial analysts and compliance reviewers by improving consistency and reducing repetitive errors. Over time, cumulative gains (greater gains at later occurrences) suggest growing returns to deploying persistent agents across repeated business processes.
  • However, organizational capture of these gains requires investment in feedback channels (expert rubrics), integration of reflection/storage mechanisms, and compute budgets—factors that will shape adoption speed and distribution of benefits.
• Research and procurement recommendations for economists and policymakers
  • Use paired longitudinal benchmarks (like FinEvo-Bench) when estimating productivity improvement or displacement risk — they better capture cumulative learning than cross-sectional task suites.
  • When evaluating vendor claims about "self-improving" systems, require controlled within-organization tests with paired baselines and structured feedback to attribute gains to retained experience.
  • Consider cost-per-point (rubric-score) improvement and cost-per-avoided-compliance-issue as practical ROI metrics. Translate compliance-issue reductions into expected monetary savings to compare against operational costs (compute, integration, monitoring).
  • Monitor for overfitting or unsafe generalization; prefer designs that emphasize skill distillation and rubric-based feedback over raw answer replay to reduce risk of inappropriate deployment.

Overall, FinEvo-Bench supplies a practical methodology and dataset for measuring the economic value of persistent learning in professional AI agents, highlighting important trade-offs (performance, compliance, cost) that matter for adoption and regulation in financial services.