Language-model research agents are effective engineering optimizers but not autonomous inventors: they routinely translate directions into working improvements yet seldom produce novel methods, and practical value depends more on consistency, experience transfer, and harness design than on peak performance.

Main Finding

Current frontier agents can perform effective, long-horizon engineering-style R&D (finding and implementing useful improvements), but they are not yet autonomous scientific innovators. Performance is shaped by three interacting components — the underlying model, how it reuses accumulated experience, and the surrounding harness — so final scores alone hide where progress is gained or lost, how repeatable results are, and whether experience transfer helps or harms.

Key Points

• Evaluation framework: decomposes the research loop into three deterministic, rule-based process metrics:
  • C1 Solution Framing — how quickly/early an agent discovers useful directions.
  • C2 Execution — whether proposed changes are translated into runnable/correct artifacts.
  • C3 Feedback Control — whether agents retain recoverable gains and correct regressions.
• Experimental scope:
  • 7 frontier models: Claude-Opus-4.7, GPT-5.5, Gemini-3.1-Pro, GLM-5.2, Kimi-K2.7-Code, DeepSeek-V4-Pro, LongCat-2.0.
  • 36 long-horizon tasks (AutoLab) across 4 families: Model Development (7), System Optimization (15), Puzzle & Challenge (10), CUDA (4).
  • 3 independent rollouts per model–task (756 rollouts); metrics reported as avg@3 (typical) and best@3 (ceiling).
• High-level outcome and cost results:
  • Opus-4.7 leads on avg@3 = 0.739 and best@3 = 0.790, but at much higher mean inference cost ($89.9/task).
  • GPT-5.5 and GLM-5.2 deliver similar performance at far lower costs (GPT ≈ $16.5, GLM ≈ $33.0 per task).
  • Low-cost models (LongCat, DeepSeek) trade some performance for very low cost ($3.9–$4.3/task).
• Process-level patterns:
  • Execution (C2) is generally high and compressed across models (0.880–0.967), meaning most agents can deliver runnable solutions.
  • Solution Framing (C1, 0.473–0.612) and Feedback Control (C3, 0.772–0.928) show much more variation — these are the primary differentiators.
  • Similar final outcomes can arise from different process profiles (e.g., GPT-5.5: stronger Execution but weaker Feedback Control vs Gemini-3.1-Pro: opposite).
• Task-specific bottlenecks:
  • CUDA tasks are the hardest for discovery and execution (low C1 and C2) but have high C3.
  • Model Development tasks show the opposite: execution is easy (high C2) but stabilizing gains is hard (low C3).
  • Puzzle & Challenge tasks are easiest overall.
• Experience reuse and harness effects:
  • Accumulated intra-task experience usually helps preserve discoveries but can also propagate misleading conclusions (positive and negative transfer both observed).
  • Inter-task transfer can materially change rankings: e.g., transferred experience raised DeepSeek-V4-Pro’s avg@3 by +0.093 but lowered Gemini-3.1-Pro’s by −0.017.
  • Harness choice mainly affects reliability/stability (run-to-run variance) rather than best achievable scores; native harnesses sometimes yield more stable rollouts than a shared harness.
• Novelty:
  • Genuine methodological novelty is rare: only 3 of 252 best-seed solutions qualified as novel under their LLM-judged review protocol.
• Consistency vs ceiling:
  • Average performance separates models more than peak performance (highest-to-lowest gap 0.237 under avg@3 vs 0.122 under best@3), indicating headroom in improving reliability and selection.

Data & Methods

• Tasks: 36 long-horizon R&D tasks from AutoLab, each providing an objective, suboptimal starting artifact, an expert reference, a verifier, and a wall-clock budget (2–12 hours).
• Models: seven state-of-the-art commercial/research LLMs (listed above) evaluated in a controlled setup.
• Harness: primary cross-model comparison used a shared harness (Claude Code); additional experiments compared model-native and open-source harnesses to test harness effects.
• Rollouts: 3 independent rollouts per model–task pair (756 total).
• Outcome metrics: avg@3 (mean over 3 rollouts) and best@3 (max over 3 rollouts).
• Process metrics: C1, C2, C3 computed deterministically from verifier outcomes and recorded trajectory signals (no LLM judgments used for the process metrics).
• Experience meta-capability: measured via controlled intra-task (within-task continued runs) and inter-task (transfer across tasks) comparisons to estimate how stored experience affects later decisions (Mintra, Minter).
• Cost accounting: token consumption and wall-clock time recorded; mean inference cost estimated using public API prices.
• Scale: evaluation required roughly $100K of model inference.

Implications for AI Economics

• Cost-performance tradeoffs: There is an explicit tradeoff between per-task inference cost and reliability/peak performance. Firms and labs must weigh marginal gains (higher best@3) versus operational costs — lower-cost models can be attractive when budget-constrained, while high-cost models yield higher ceilings.
• Value of systems and integration: Substantial performance differences arise from harness and experience-management design, not just backbone models. Economic value therefore attaches to system engineering (harnesses, experience stores, rollout-selection strategies) as much as to model weights. This favors firms that can integrate models into robust agent systems.
• Returns to investment in stability vs peak capability: The larger dispersion in avg@3 than best@3 implies high returns to investments that increase reliability (e.g., ensemble/selection, harness stabilization, rollout-aware training), especially for use cases requiring predictable output rather than occasional breakthroughs.
• Impact on R&D productivity measurement: Agent outputs can improve artifacts but tend to be incremental/optimizing rather than novel. Economic assessments of AI-driven R&D should separate gains from engineering optimization (productivity/efficiency improvements) from genuine innovation (novel methods offering broader spillovers).
• Market structure and competitive advantage: Because experience-transfer mechanics and harness design can reorder performance, organizations that develop superior experience management, transfer protocols, and harnesses may obtain outsized competitive advantages beyond raw model access.
• Risk and policy relevance: Agents are not yet autonomously discovering new methodologies often, but experience accumulation can produce positive or negative transfer and sometimes amplify lock-in to suboptimal directions. Economic and regulatory analyses of AI-driven self-improvement and systemic risk should therefore consider closed-loop learning dynamics and system-level design, not only per-model capability.
• Investment priorities for deployers:
  • Prioritize engineering effort on harness design and experience-store mechanisms for predictable R&D automation value.
  • Allocate budget for strategies that improve run-to-run reliability (inference-time selection, rollout-based training), which yield high expected returns given large avg vs best gaps.
  • For spillover/novelty-focused objectives (e.g., breakthrough research), current agents are unlikely to substitute human creativity broadly; policy and corporate R&D strategies should treat these agents as productivity multipliers rather than autonomous innovators.

If you want, I can: - Extract the full numeric table of model-by-metric scores and costs, - Map these findings into a simple ROI model for adopting an agent in corporate R&D (cost vs expected performance/stability), - Or draft policy-relevant recommendations for regulators and funders.