An autonomous evolutionary system, ASI-Evolve, discovered hundreds of improved models and pipelines—finding 105 state-of-the-art linear-attention architectures and yielding large benchmark gains (e.g., +18 MMLU, up to +12.5 RL points)—offering early evidence that AI can speed parts of the AI research loop.

Main Finding

ASI-EVOLVE is an agentic "AI-for-AI" framework that closes a learn–design–experiment–analyze loop by (1) injecting accumulated human priors via a cognition base and (2) distilling experimental outcomes via a dedicated analyzer. In a unified set of large-scale experiments, ASI-EVOLVE autonomously produced meaningful advances across three foundational AI development components—model architectures, pretraining data curation, and training algorithms—providing initial evidence that closed-loop AI research can accelerate AI progress. The codebase is open-sourced: https://github.com/GAIR-NLP/ASI-Evolve.

Key Points

• Core design:
  • Iterative pipeline: sample context from a persistent database, retrieve cognition priors, generate candidate programs (Researcher), execute experiments with early-rejection and debugging (Engineer), and distill multi-dimensional outputs into compact reports (Analyzer).
  • Persistent stores: Database D (history of nodes = motivation, code, results, analysis, metadata) and Cognition C (task-relevant human priors indexed by embeddings).
  • Sampling policies supported: UCB1, greedy, random, MAP-Elites/island.
• Two central innovations:
  • Cognition base: injects curated human literature/heuristics to steer exploration and accelerate cold-start performance.
  • Analyzer: converts verbose experimental logs and multi-metric outputs into reusable, structured insights for future rounds.
• Empirical achievements (selected highlights):
  • Neural architecture design (linear/efficient attention): discovered 1,350 candidates across 1,773 rounds, 105 architectures surpass human-designed DeltaNet; best model +0.97 points above DeltaNet (nearly 3× gains of recent human tweaks).
  • Pretraining data curation: evolved pipelines improved average benchmark performance by +3.96 points; MMLU gains exceeded +18 points.
  • Reinforcement learning algorithm design: discovered algorithms outperform GRPO by up to +12.5 (AMC32), +11.67 (AIME24), and +5.04 (OlympiadBench).
  • Cross-domain transfer: initial evidence in mathematics and biomedicine (drug–target interaction: +6.94 AUROC in cold-start).
• Ablations and diagnostics:
  • Cognition base substantially improves cold-start climb speed and iteration efficiency.
  • Choice of sampling strategy strongly affects long-run trajectory and sustained improvement.
  • Engineering guards (static checks, debug agent, novelty filter) are important for constraint-heavy large-codebase evolution.

Data & Methods

• Conceptual framing:
  • Introduces Ltask = ⟨Cexec, Sspace, Dfeedback⟩ to characterize tasks by execution cost, search-space complexity, and feedback complexity, motivating architecture/design choices for long-horizon, high-cost scientific automation.
• System modules:
  • Researcher: LLM- or diff-based program generation conditioned on sampled database nodes + retrieved cognition entries.
  • Engineer: runs full evaluation scripts; supports early rejection (wall-clock timeouts, quick tests) and optional LLM-based quality judging.
  • Analyzer: ingests raw logs, training dynamics, per-benchmark breakdowns, and produces compact causal analyses that are stored and used for retrieval.
  • Cognition store: curated human prior corpus (e.g., ~150 entries from 100 papers for the linear-attention task) used via embedding search to guide proposals.
• Experimental workflow specifics (example: model architecture task):
  • Candidate generation with static checks (complexity bounds, chunk-wise structure, causal masks), debug agent for runtime fixes, and novelty checks to avoid duplicates.
  • Multi-stage evaluation: quick/cheaper exploration stage (small models ~20M params trained for short steps on subset), then more expensive validation of promising candidates. (Paper gives concrete benchmarks, training steps, and evaluation splits per task.)
• Benchmarks and metrics:
  • Architecture: evaluated on a suite of language-modeling and sequence benchmarks (10 core benchmarks at exploration stage).
  • Data curation: measured via downstream benchmark suites including MMLU.
  • RL algorithms: evaluated on AMC32, AIME24, OlympiadBench.
• Storage and selection:
  • Database retains top candidate pool (e.g., top-50) and samples contexts from the high-performing set to encourage progressive improvement.

Implications for AI Economics

• Productivity and R&D efficiency:
  • If generalizable, ASI-EVOLVE-style systems could raise marginal productivity of AI R&D by automating parts of hypothesis generation, code modification, experiment orchestration, and insight distillation—reducing human time per iteration and accelerating iteration velocity.
  • The cognition base plus analyzer model suggests AI agents can internalize and accumulate institutional memory, lowering the cost of rediscovery and improving long-run returns to AI R&D investments.
• Capital vs. labor effects:
  • High computational costs remain (many GPU-hours per candidate), so gains primarily increase returns to capital (compute infrastructure, cloud/GPU investments) and algorithmic automation, potentially substituting some human experimental labor (junior engineers, routine architecture search tasks) while increasing demand for higher-skill oversight roles (tooling, validation, safety).
• Returns to scale and market concentration:
  • Organizations with large compute budgets and engineering pipelines can deploy closed-loop search at scale, possibly increasing first-mover advantages and market concentration. Open-sourcing mitigates but does not fully offset capital asymmetries.
• Innovation dynamics and creative complementarity:
  • ASI-EVOLVE shows AI can contribute non-trivial innovations (novel architectures, algorithms, data pipelines). This implies complementarity between humans (setting objectives, high-level theory, ethical oversight) and automated agents (search, experimentation), potentially shifting human roles toward meta-science, evaluation, and translational work.
• Implications for R&D budgeting and policy:
  • Budget models should account for higher upfront compute spending to enable automated search loops with potentially faster payoff horizons.
  • Public funding and policy may need to support shared compute and benchmarking infrastructure to democratize access and limit concentration.
• Risks and caveats:
  • Results are promising but preliminary: replication, robustness, and evaluation of long-run generalization across more domains and against stronger baselines are needed.
  • Accelerated discovery also raises risks: faster model cycles could shorten product lifecycles, raise externalities (e.g., unforeseen capabilities), and stress governance regimes.
  • High compute demand emphasizes energy and environmental cost considerations; economic assessments must include these externalities.
• Research economics questions opened by this work:
  • How do returns to compute vs. human labor evolve if closed-loop agents scale?
  • What is the impact on the allocation of funding across basic vs. applied ML research?
  • How will intellectual property, publication, and reproducibility norms adapt when agents generate substantial parts of technical progress?

Caveats: ASI-EVOLVE demonstrates strong results on several tasks, but deployment and economic impact depend on generalization beyond the reported domains, reproducibility of gains under different compute constraints, and the broader organizational adoption of such closed-loop systems.