LLM agents can run and debug training pipelines effectively but rarely change their initial training strategy, so extra experience, human guidance or more compute mainly yields local fixes rather than strategic improvements.

Main Finding

Current LLM agents are competent at executing and iterating within a chosen post-training pipeline (execution-level capability) but systematically fail to change their high-level training strategy once a run begins (strategy-level capability). The initial strategy choice—determined more by the agent’s prior/configuration than by the task—sets a hard ceiling on improvements. What is missing is a mechanism that spontaneously triggers strategy-level reevaluation during execution.

Key Points

• Two-level framework
  • Execution-level: build/repair pipelines, tune hyperparameters, debug, evaluate checkpoints.
  • Strategy-level: choose or switch the high-level training paradigm (e.g., SFT vs PEFT vs RL), stage structure, or data regime.
• Large-scale trajectory analysis (no reruns) shows:
  • Agents routinely complete full training pipelines and make meaningful within-strategy improvements.
  • Strategy lock-in: the high-level strategy is chosen early (pre-execution) and rarely changes thereafter. Only ~2.1% of adjacent training pairs probed an alternative strategy.
  • Defaults track the agent type, not the task: e.g., Claude Code trajectories anchor on full-parameter SFT (≈80.7%), while Codex CLI trajectories anchor on PEFT (≈89.6%).
• Execution competence (examples & stats):
  • Across 1,338 trajectories: average 3.82 trainings and 13.80 evaluations per trajectory.
  • Observed overall score improvement from base → submitted: 10.41% → 23.0% (varies by benchmark).
• Three hypotheses tested with escalating interventions:
• Missing experience → provided an experience-driven scaffold (experiment journal, skill library, evaluator agent). - Result: strong execution improvements (e.g., +12.6 points GSM8K; +40.8 points HumanEval) but no substantial increase in strategy switching.
• Missing guidance → human reviewer steers initial strategy. - Result: human guidance successfully redirects the initial strategy, but once training starts the agent falls back into local adjustment loops and rarely adopts suggested strategy-level changes.
• Insufficient reasoning compute → allowed much larger inference budgets (2–8× tokens). - Result: extra compute yields gains on easier tasks but almost no benefit on the hardest benchmark (AIME), and did not induce spontaneous strategy shifts.
• Behavioral asymmetry: agents reliably adopt execution-level suggestions from an evaluator but repeatedly decline strategy-level suggestions (e.g., suggestions to add an SFT warm-up or switch training paradigm were not adopted despite repeated recommendations).

Data & Methods

• Dataset:
  • 1,338 publicly released agent post-training trajectories from PostTrainBench (Rank et al., 2026).
  • Covered 7 benchmarks (math, code, writing, Q&A), 4 base models (1.7B–4B), 20 agent configurations and five scaffolds (including Claude Code, Codex CLI).
  • Each trajectory reconstructed commands, edits, training jobs, evaluations under a 10-hour budget on one NVIDIA H100 80GB.
• Annotation & counting:
  • A "training experiment" counted only when a command launched a model parameter update. Writing scripts, installing packages, or saving checkpoints without parameter updates were not separate experiments.
  • A strategy change required altering the training paradigm, data-source type, or stage structure. Hyperparameter tuning or formatting counted as execution-level changes.
  • Trajectories annotated by an LLM and reviewed by authors.
• Evaluation:
  • Standard pass@1 accuracy for most benchmarks; AIME evaluated with pass@8 due to small sample size (30 problems).
• Controlled interventions and reruns:
  • Controlled experiments used Qwen3-1.7B-Base and three benchmarks of increasing difficulty: GSM8K, HumanEval, AIME 2025.
  • Autonomous baselines: Claude Code (Opus 4.6 & GLM-5.2) and Codex CLI (GPT-5.2).
  • Experience-driven framework components:
    • Experiment journal (persistent record of plans/results/lessons),
    • Skill library (distilled training recipes & implementation knowledge),
    • Evaluator agent (analyzes outputs, diagnoses issues, suggests concrete actions).
  • Controlled runs: 3 independent runs per configuration under 10-hour budgets on four NVIDIA A800 GPUs.

Implications for AI Economics

• Value of initial strategy > marginal iterative compute:
  • Returns to post-training compute/time are heavily front-loaded into whether the initial strategy is appropriate. Additional execution-focused compute yields diminishing returns if the high-level strategy is wrong.
  • Economic implication: reallocating R&D budget from repeated within-strategy compute to mechanisms that improve initial strategy selection (or enable mid-run strategy switching) could raise ROI.
• Product and service market impacts:
  • Demand likely grows for tooling/agents that provide robust strategy-level capabilities (meta-evaluation, automated paradigm switching, exploration incentives).
  • Firms offering “smart experiment design” or human-in-the-loop strategy advisors may capture outsized value relative to pure compute providers.
• Labor & organizational design:
  • Human expertise remains valuable in the early planning/pre-execution window; organizations may optimize workflows to front-load human or higher-level agent review before long runs.
  • Conversely, manual oversight after training starts yields limited returns unless it can trigger true strategy changes.
• R&D and capital allocation:
  • Investments in meta-reasoning systems (agents that can spontaneously reevaluate strategy, estimate switching costs vs expected improvement, or meta-learn when to explore) may be more productive than scaling execution compute.
  • Resource allocation models for AI R&D should incorporate high switching costs and asymmetric benefits of strategy change; cost-benefit models must account for the empirical rarity of strategy updates.
• Policy, safety, and alignment:
  • Strategy lock-in raises a failure mode where a mis-specified or biased initial strategy persists despite evidence—relevant for safety oversight and audits.
  • Regulatory or audit mechanisms might focus on early-stage strategy validation, or require agents to implement explicit reassessment triggers.
• Research agenda:
  • Design economic incentives and algorithmic primitives for endogenously triggered strategy shifts (meta-controllers, exploration bonuses that overcome sunk-cost bias, structured experiment comparison protocols).
  • Evaluate mechanisms for cost-effective strategy comparison (e.g., cheap probes that estimate expected marginal gain of a switch) to overcome the empirical reluctance to switch.

Limitations to bear in mind - Primary trajectory analysis is observational over publicly released runs (no reruns for those data); controlled experiments are narrower in model and benchmark scope. - Annotations used LLM labeling with author review—there may be labeling or interpretation limits. - Generalization to substantially different agent architectures, budgets, or future agent designs requires empirical validation.

Bottom line: improving automated AI R&D economically and technically likely requires solving the meta-level problem—mechanisms that prompt, justify, and execute strategy-level changes—rather than only scaling execution competence or inference compute.