LabAgent turns a lab’s public code and papers into reusable, executable skills and reproduces many computational-biology results more reliably than two commercial agent harnesses; it often reaches published metrics (32/65 ADMET entries within reported SD) but does so at higher per-task cost.

Main Finding

LabAgent is a multi-agent framework that converts a laboratory’s public artifacts (papers, code, issues, tutorials) into a persistent, executable skill library and an experience-conditioned reproduce agent. By (1) extracting and validating “skills” that capture the procedural tacit knowledge of methods, (2) recording execution traces, diagnoses and fixes in persistent memory, and (3) assembling role-specialized agents to plan/implement/run/verify reproductions, LabAgent both reproduces published computational biology results and uses the validated methods to drive new discovery. Across four life-science domains it outperforms two commercial agent harnesses (Claude Code CLI on Opus 4.8 and Codex CLI on GPT-5.5) on reproducibility and analysis benchmarks, at a modestly higher operational cost.

Key Points

• Architecture
  • Explore agent: mines a lab’s public artifacts, grounds descriptions to code, synthesizes executable skills (including undocumented preprocessing, pinned versions, failure modes).
  • Reproduce agent: assembles role-specialized subagents (Probe, Code, Env/Data, Runner, Verifier, Analyst), reuses prior lessons from persistent memory, runs and verifies against the original study metrics, localizes and repairs failures, writes back fixes and logs.
  • Persistent skill & reproduction memory: stores skills, evidence grounding, execution logs, diagnoses (failures→fixes), and verified results so later reproductions can be experience-conditioned rather than starting fresh.
  • Skill admission policy: a skill only enters the library after running end-to-end successfully.
• Empirical performance (selected results)
  • ADMET drug-property leaderboard reproductions (Therapeutics Data Commons):
  • Rebuilt 65/66 target entries (one upstream repo deleted).
  • Landed within the published standard deviation on 32/65 entries — more than the two commercial harnesses.
  • Mean rank across entries: LabAgent 2.62 (best among agents); leaderboard ground truth mean rank 1.87.
  • Example trace: rebuilding MiniMol required iterative fixes (deps, pinning, shared-object issues) and achieved AUPRC 0.455 vs published 0.474 ± 0.025.
  • Cost: US$4.56 per entry (higher than comparators), with higher spend tied to iterative debugging and dependency resolution that produced durable fixes.
  • Biomedical data-analysis benchmarks:
  • BiomniBench-DA: LabAgent mean rubric score 74.4 (lead of +2.1 vs Claude Code, +8.7 vs Codex). Ablation without the explore agent scored 71.6 (shows importance of the skill library).
  • GeneBench-Pro: passed 2/10 public problems (Claude Code 1/10, Codex 0/10). Partial-credit averages: LabAgent 31.4, Claude Code 28.0, Codex 16.2. Cost ≈ US$2.43/problem.
  • Example domain insight: in a cis-eQTL single-cell task LabAgent reconstructed activated monocyte subpopulation and ultimately recovered an effect estimate close to the reference; discrepancies traced to downstream handling of a donor-level nuisance variable.
  • Additional domains: applied to protein variant effect prediction and statistical genetics; LabAgent reconstructed a published figure even when no figure code was released, and made departures locatable and named in memory.
• Comparative baseline
  • Baselines: Claude Code CLI (Opus 4.8) and Codex CLI (GPT-5.5). LabAgent typically yields better reproducibility & process scores but at higher monetary cost and with variability across task types.
• Limitations observed
  • Higher per-run cost due to iterative execution and debugging.
  • Not all tasks reproduced perfectly; some domains and cases still fail or return partial credit.
  • Relies on the availability and completeness of public artifacts (papers, code, issues).

Data & Methods

• Data sources and domains
  • Therapeutics Data Commons ADMET leaderboards (22 datasets; 65 reproducible entries used).
  • BiomniBench-DA (50 tasks spanning 16/17 task types; rubric scores process decisions).
  • GeneBench-Pro (10 public problems; end-point estimand scoring).
  • Additional case studies in protein variant effect prediction and statistical fine-mapping (including reconstruction of a published figure without released code).
• Skill generation
  • Two routes: repository mode (target names repo) and task-type mode (agent searches for code implementing method).
  • Draft skills are synthesized and then validated by running end-to-end; only passing skills are stored.
• Reproduction workflow
  • The reproduce agent loads the best-matching skill, reads prior execution experience from memory (commands/fixes), then executes a multi-role iterative pipeline that plans, edits, runs, diagnoses failures, and retries until verifier metrics match published values or a budget is exhausted.
  • Verification is scientific: success requires reproducing reported metrics (not merely finishing without error).
• Baselines and costing
  • Compared to two commercial harnesses (Claude Code CLI, Codex CLI). Costs reported per-task (USD) reflect compute and model usage during runs.
• Metrics reported
  • Reproducibility: within published standard deviation, mean rank on leaderboard entries.
  • Process quality: rubric scores on BiomniBench-DA.
  • Outcome accuracy: pass/partial/failed on GeneBench-Pro and specific numerical closeness to published estimates.
  • Operational cost per run/task.

Implications for AI Economics

• Productivity and R&D efficiency
  • LabAgent formalizes and preserves tacit experimental and computational knowledge as executable assets; this can lower the recurring cost of method handoff and reduce repeated debugging across labs, increasing effective R&D throughput and reducing waste from lost institutional knowledge.
  • The approach turns reproducibility work into a durable public good (executable skills + fixes), potentially raising the marginal productivity of downstream researchers who reuse validated workflows.
• Labor demand and skill composition
  • Automation of reproduction and method maintenance may substitute for some experienced research-engineer time (dependency debugging, environment setup), shifting human labor toward higher-level experimental design and interpretation.
  • Demand rises for engineers who can curate, audit, and extend agent-generated skills; also for domain experts to validate scientific verification criteria and edge-case decisions.
• Cost structure and business models
  • Lab-specific agent creation has higher up-front and per-run compute cost than some general commercial harnesses, but produces persistent knowledge that amortizes over subsequent reproductions and discovery tasks—favorable returns as reuse scales (economies of scope).
  • Potential services/products: lab-specific agent-as-a-service, reproducibility-as-a-service for publishers/funders, and marketplaces for verified skill libraries. Pricing models could reflect upfront skill extraction + pay-per-execution.
• Incentives and public-good provision
  • Incentivizing labs to publish richer artifacts (pinned envs, issue histories, runnable demos) increases the agent’s effectiveness; funders and journals could accelerate adoption by requiring artifact quality standards.
  • A standardized format for skill packaging and execution logs would lower transaction costs in reuse and enable easier valuation of reproducibility investments.
• Measurement and evaluation
  • Economists can model the value of LabAgent-style systems by estimating reduced replication costs, increased rate of valid follow-on research, and time saved per researcher; these inputs feed return-on-investment calculations for adopting agent technologies in research institutions.
• Risks, market concentration, and governance
  • Centralized provider dominance (few LLM/agent vendors) combined with labs outsourcing reproducibility could create lock-in and concentration risks; open standards and interoperability are important.
  • Incorrect skill synthesis or silent failures introduce scientific risks—necessitating audits, provenance, and human-in-the-loop verification for high-stakes research.
• Policy and regulatory considerations
  • Funders or journals might require machine-verifiable reproducibility artifacts; otherwise, inequities could grow between well-resourced labs that can pay for agent curation and under-resourced groups.
  • Intellectual property and data/privacy rules matter: agents ingest public artifacts, but deploying agent-derived pipelines on private or sensitive datasets raises compliance complexity.
• Research-agenda suggestions for AI economists
  • Quantify productivity gains from experience-conditioned skill reuse across labs and over time.
  • Model labor reallocation effects: how much and which research roles shrink/grow with agent adoption.
  • Analyze market structures for lab-specific agent services (competition, pricing, and welfare effects).
  • Evaluate social returns from mandating artifact standards that facilitate agent extraction of skills.

If you want, I can (a) extract a concise list of the quantitative results (tables of scores/costs) from the paper for economic modeling, or (b) sketch a simple economic model to estimate ROI of deploying LabAgent in a university lab. Which would you prefer?