Curated shared knowledge, not bespoke agents, can drive cheaper, more transferable self-improvement: across reasoning, coding and terminal benchmarks, agents that learn by contributing to a persistent knowledge base solved more problems at lower cost than agent-centric alternatives, and the distilled insights boosted performance on held-out tasks and other LLM families.

Main Finding

Shifting self-improvement from persistent agent modifications to a curated, persistent knowledge base (knowledge-centric self-improvement, KSI) yields higher task solve rates and lower dollar cost than strong agent-centric baselines. Distilled, evidence-grounded knowledge bundles accumulated via a simple forum+distillation protocol transfer to held-out tasks and across LLM families, indicating reusable, portable improvement that is separable from any single agent instance.

Key Points

• Paradigm: Agents are generic and disposable workers; the only persistent object is a curated knowledge base that agents read from and write to.
• Protocol (three stages):
• Task-level forum — agents post evidence-grounded local claims (what worked/failed, concrete checks/hypotheses) citing prior posts.
• Cross-task forum — agents surface recurring patterns, support/challenge claims across tasks.
• Distillation — an LLM selects and composes actionable, scoped bundles (transferable insights, confirmed constraints, rejected hypotheses, checks, next steps) that seed future agents.
• Evidence-handling: Disagreement is treated as evidence; the protocol explicitly preserves competing hypotheses and only distills claims that remain actionable and supported.
• Benchmarks: Evaluated on abstract reasoning (ARC-AGI-1/2), coding (Polyglot, SWE-bench Pro), and terminal skills (Terminal-Bench 2).
• Baselines: Compared against agent-centric self-improvement systems (Darwin Gödel Machine, HyperAgents, Meta-Harness) and prompt-optimization approaches.
• Main quantitative outcomes (Haiku 4.5 results):
  • ARC-AGI-1: OURS 86.7% ±4.2 solve, cost ≈ $76 ±16 vs HyperAgents 70% ($234).
  • ARC-AGI-2: OURS 82.7% ±6.1, cost ≈ $80 ±1 vs HyperAgents 60% ($188).
  • Polyglot: OURS 68.0% ±2.0, cost ≈ $126 ±6 vs HyperAgents 52% ($190), DGM 58% ($281).
  • SWE-bench Pro: OURS 64.0% ±2.0, cost ≈ $208 ±19 vs HyperAgents 42% ($431), DGM 54% ($713).
  • Terminal-Bench 2: OURS 43.8% ±3.4 solve vs best baseline Meta-Harness 37.6%.
• Transfer: Distilled bundles improved zero-shot performance on held-out tasks and remained effective when consumed by different LLM families — evidence that gains are not run- or LLM-specific.
• Implementation notes: Generation-based evaluation (50-task pools, 10 generations), agents given standard toolset where applicable (file I/O, shell, test runs), knowledge base stores attempt table, forum posts, and distilled bundles.

Data & Methods

• Experimental design:
  • Tasks: 50-task subsets sampled for each benchmark (coding, abstract reasoning); Terminal-Bench 2 used 89 real environments.
  • Generations: At each generation, one fresh agent per unsolved task reads distilled per-task and cross-task bundles, attempts the task once (with standard tool access where relevant), then posts attempt evidence to task forums.
  • Curation: Task-level posts (local evidence) → cross-task discussion (support/challenge) → distillation LLM produces compact bundles for next generation.
  • Persistence: Only the knowledge base evolves; agents are re-instantiated with clean contexts each attempt.
• LLMs and tooling:
  • Main LLM for comparisons: Haiku 4.5 (Anthropic stack). Also experimented across LLM families (e.g., GPT-5.4-mini) to test transfer.
  • Agent tooling for coding/terminal tasks: file read/write, shell execution, test invocation; baselines rerun under the same tool set for fairness.
• Evaluation:
  • Metrics: Solve rate (task success) and dollar cost per benchmark (USD cost of LLM calls).
  • Seeds: OURS reported over 3 seeds; baselines rerun under the same evaluation protocol when possible.
• Protocol artifacts:
  • Attempt table: records each attempt & outcome.
  • Forums: threaded posts with typed schemas (fields like hypotheses, checks, constraints).
  • Distilled bundles: typed fields (transferable insights, constraints, pitfalls, checks, next steps).
• Controlled comparisons: The design isolates knowledge curation as the sole improvement mechanism (no agent specialization, private memory, or prompt-walkover across generations).

Implications for AI Economics

• Assetization of Knowledge vs Model Lock-in:
  • KSI reframes self-improvement as creation of a persistent knowledge asset (distilled bundles) rather than continual agent specialization or model modification. This reduces vendor/architecture lock-in because improvements are external, inspectable, and transferable across agent implementations and LLM families.
• Lower marginal cost of improvement:
  • Empirically lower dollar cost per performance gain suggests investing in curation infrastructure (processes, human oversight tooling, knowledge-distillation LLMs) can be more cost-effective than repeated expensive agent retraining or heavy agent engineering.
• Economies of scale and reusability:
  • Curated knowledge is reusable across many fresh agent instances and tasks; once distilled, a single bundle can amortize across downstream uses, improving returns on the curation investment.
• Market and product implications:
  • Distilled knowledge bundles are a potential product—tradeable, auditable artifacts that could be licensed, shared as public goods, or monetized by firms. This could create marketplaces for high-quality distilled task knowledge, shifting competition toward curation quality and governance.
• Organizational design and labor:
  • Firms may reallocate resources from customizing/maintaining complex agents toward building curation pipelines, verification processes, and knowledge governance. The role of human experts may shift to supervising distillation, resolving contentious claims, and auditing bundles.
• Interoperability & competition:
  • Portability of distilled knowledge reduces switching costs and could intensify competition, as smaller teams can reuse high-quality curated knowledge with off-the-shelf agents.
• Risks, caveats, and open questions (economic considerations):
  • Generality limits: evidence is from specific benchmarks; real-world heterogeneity might reduce transferability or require larger curation budgets.
  • Quality assurance and governance: curated knowledge must be verified and maintained; poor distillation could propagate harmful or incorrect guidance at scale.
  • Upfront and ongoing costs: building reliable forums, verification processes, and distillation models requires investment; the cost-benefit depends on task reuse frequency and value.
  • Potential for acute competition on curated knowledge ownership and lock-in via proprietary bundles — legal, ethical, and market governance issues arise.
• Strategic takeaway:
  • From an AI-economic perspective, investing in modular, shareable knowledge curation infrastructure can yield higher ROI than continually engineering monolithic, specialized agents—especially where tasks are heterogeneous and reusable guidance exists. Policymakers and firms should consider incentives and standards for quality, provenance, and marketplace design for distilled knowledge assets.