AI agents can execute many end-to-end computational-biology analyses but remain far from reliable: the best models reproduce about 48% of published artifacts, struggle on very large datasets and long multi-step pipelines, and the highest performers are often cheaper and more token-efficient than poorer models.

Main Finding

BixBench3 evaluates whether LLM-based agents can execute end-to-end, research-study–scale computational-biology analyses from raw data to published-style artifacts. Across 20 real-study tasks (138 graded artifacts), frontier models reproduced at best ~48% of requested artifacts closely enough to preserve their main scientific interpretation. Performance, cost, and token/runtime usage vary widely across models, with clear trade-offs: some models achieve near-top accuracy at substantially lower cost and token usage than others, while large raw datasets and long multi-step analysis chains substantially reduce agent success.

Key Points

• Benchmark scope and scale
  • 20 tasks drawn from published computational-biology studies spanning 17 assay types and 9 domains.
  • 138 graded artifacts (56 depth-1, 44 depth-2, 38 depth-3+); average raw input per task = 67 GB (range 7–241 GB).
  • 260 completed model runs and 1,794 model–artifact evaluations across 13 modern LLMs.
• Top-line performance
  • Best model: GPT 5.6 Sol (mean task score 0.48). Other high performers: Kimi K3 (0.47), GLM 5.2 (0.46), Claude Opus 4.8 (0.46).
  • Worst reported in the set: Gemini 3.1 Flash Lite (0.00).
  • Artifact pass rates by depth (averaged across models): depth-1 = 0.30, depth-2 = 0.44, depth-3+ = 0.24.
• Data scale and analysis depth effects
  • Tasks with >100 GB raw input: mean score ≈ 0.10; tasks <50 GB: ≈ 0.37.
  • Performance degrades with longer dependency chains (depth 3+ artifacts are hardest).
• Cost, tokens, and runtime
  • Mean per-task consumption: 102 million tokens, 6.8 hours wall time, 695 model turns.
  • Mean cost per task ≈ $43, but model costs varied 367× (min ≈ $0.35, max ≈ $129.14 observed).
  • Longest single attempts reached 24 hours, 1.07 billion tokens, and $525.
  • Higher cost/longer runs did not guarantee higher accuracy; a Pareto frontier of cost–performance/efficiency exists (some cheaper models are near-optimal).
• Failure modes and diagnostics
  • An LLM judge annotated failure modes per run; total failure-tag counts strongly negatively correlated with task score (Spearman ρ = −0.92).
  • Common failure modes associated with low scores: premature termination, repetitive retry loops, environment setup failures, incomplete data outputs, and synthetic/placeholder outputs.
• Benchmark constraints
  • Tasks prescribe methods and outputs (the agent executes methods; the benchmark does not test autonomous selection of research questions or methods).
  • Programmatic grading compares produced artifacts to published artifacts; pass threshold was calibrated by domain experts (artifact score ≥ 0.80 considered preserving main interpretation).

Data & Methods

• Task construction
  • Each task prompt included: workspace instructions, the research objective (question + raw/reference data), method guidance (tools, parameters, contrasts, filters mirroring the paper), and exact required output file paths/formats.
  • Tasks represent the original paper’s analysis DAG; agent outputs graded at artifact level and averaged to task scores.
• Runtime environment
  • Agents executed in Docker on Google Cloud n2-standard-32 VMs (32 vCPUs, 128 GB RAM), 500 GB boot disk, no GPU; 24-hour wall-clock cap; no fixed token limit.
  • Preinstalled scientific stack (Python/R/Java/etc.); agents could install additional software during runs.
  • Network access controlled via an allowlist and an access-request tool.
• Models and evaluation
  • 13 frontier LLMs evaluated (names reported in paper; top models listed above).
  • Programmatic artifact scoring compared agent outputs to published study artifacts; binary pass based on expert-calibrated threshold (0.80).
  • An automatic judge (LLM) assigned failure-mode tags to runs from a pre-defined set.
• Key quantitative metrics reported
  • 20 tasks, 138 artifacts, 260 runs, 1,794 artifact evaluations.
  • Average tokens per task: 102M; average time: 6.8 hours; average cost ≈ $43.
  • Model-specific variability in cost and cache-hit effects (example: GLM 5.2 observed higher cost due to lower cache hit rate; authors reported a cache-adjusted estimate).

Implications for AI Economics

• Cost–performance trade-offs and procurement
  • Buyers should evaluate models on cost-per-artifact and cost-per-task, not only top-line accuracy. Several models (e.g., Kimi K3) provide near-top performance at materially lower cost.
  • Cache behavior and deployment routing materially affect realized costs (observed in GLM 5.2). Procurement should include metrics for cache hit rates and provider routing.
• Labor substitution and productivity
  • Agents can automate substantial portions of complex data-analysis pipelines (reproducing many downstream artifacts), implying potential reductions in researcher time for prescribed analyses.
  • However, median success is far from complete automation: human oversight is still required for method choice, error recovery, and interpretation—so economic value is greatest for augmenting, accelerating, or parallelizing analysis work rather than fully replacing skilled researchers.
• Capital and infrastructure economics
  • Large raw data sizes sharply reduce agent effectiveness, implying returns to investing in (a) models with better long-context/large-data handling, and (b) specialized compute infrastructure (data staging, optimized I/O, model–tool co-location) to reduce effective latency/token usage.
  • Firms and labs should assess whether to adopt general-purpose LLMs or invest in domain-specific agents/pipelines: specialized stacks may reduce tokens/runtime and improve reliability.
• Pricing models and incentives
  • The finding that maximal performance does not require maximal spend suggests pricing strategies and tiered SLAs can be effective: mid-tier models may hit sweet spots for marginal productivity.
  • Vendors that expose predictable cost metrics (tokens, cacheability, real-world pass rates) will be easier to economically evaluate.
• R&D and investment priorities
  • Investments that improve long-context coherence, error recovery, and multi-step planning will likely yield outsized economic benefits for research automation.
  • Benchmarking frameworks like BixBench3 are valuable procurement tools—enabling buyers to measure real-world task performance, not just benchmark scores on isolated analyses.
• Risk and hidden costs
  • Failure modes (premature termination, placeholders, setup faults) imply hidden verification and remediation costs; economic models should include expected human verification hours and potential costs of incorrect scientific outputs.
  • Because BixBench3 requires prescribed methods, current agent capabilities are better at execution than at scientific decision-making—economic value is thus contingent on the quality of human-led experimental design.

Summary takeaway: LLM-based agents are approaching practical utility for executing multi-step computational-biology analyses, but partial coverage, sensitivity to data scale/depth, and heterogeneous cost-efficiency mean buyers and labs must treat deployment as an optimization problem (choose models/pipelines that best trade cost, speed, and reliability) and continue to budget for human oversight and infrastructure investments.