0 cumulative citations
View corpus contextState-of-the-art AI agents make substantial partial progress on cross-domain scientific workflows but rarely finish them: the best configurations fully completed just 20.6% of 97 evaluated tasks, even though average rubric scores approached 88%, revealing a large gap between partial progress and reliable end-to-end delivery.
FrontierChallenge: Evaluating Scientific Workflow Completion
FRONTIERCHALLENGE shows that modern agentic LLMs often make substantial partial progress on complex scientific workflows but rarely produce fully complete, reproducible deliverable bundles—top configurations fully completed only 20.6% of 97 tasks despite Avg. Scores near 88%.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated programs, or a single domain. We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific workflows. In this paper, we release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. We evaluate twelve frontier models with three agent scaffolds. Pass Rate measures the fraction of tasks satisfying the full-completion criterion, while Avg. Score captures partial progress. Each of the best-performing configurations completed only 20 of the 97 released tasks, yielding a Pass Rate of 20.6%. Partial progress translated especially poorly into complete delivery in analytical chemistry and electrochemistry/environment: Avg. Scores reached 87.6 and 94.9, but the highest Pass Rates were only 4% and 0%. Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion. These findings show that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered, highlighting the need to evaluate end-to-end workflow execution and the completeness of scientific deliverables together.
Summary
Main Finding
FRONTIERCHALLENGE exposes a large gap between partial progress and reliable end-to-end scientific delivery by current frontier models. Across 97 released cross-domain scientific workflows, the best configurations completed only 20 tasks (Pass Rate = 20.6%) despite high average partial scores (highest Avg. Score = 87.9). High partial scores and confident language-level claims of completion do not reliably predict actually satisfying a workflow’s full deliverable contract.
Key Points
- Benchmark scope
- Pool of 300 curated end-to-end scientific workflows; 97 tasks publicly released and evaluated here (74 Hard, 23 Medium).
- Six reporting domains: quantum chemistry (20), molecular dynamics (16), materials characterization (22), analytical chemistry (23), life science (10), electrochemistry/environment (6).
- Tasks require heterogeneous, mutually consistent deliverables (reports, structured tables, figures, executable code, simulation artifacts).
- Evaluation metrics and rules
- Unit of evaluation = complete artifact bundle.
- Each task has an executable Grader returning sm_i ∈ [0,100].
- Primary metric: Pass Rate = fraction of tasks with sm_i ≥ 99.9 (strict full-completion criterion after averaging three Judge runs).
- Complementary metric: Avg. Score = mean sm_i across tasks (measures partial progress; not a substitute for Pass Rate).
- Judges: GPT-5.6 Sol used to adjudicate rubric-based semantic criteria (3 runs averaged).
- Models and agent scaffolds
- Three agent scaffolds: Codex, Claude Code, Frontier Agent (Apodex Agent Team).
- Twelve frontier models tested (examples: GPT-5.6 Sol/Terra, Grok 4.6, Kimi K3, Claude Opus 5, Qwen 3.8 Max, Gemini 3.7 Flash, Apodex 1.1, DeepSeek variants, GLM-5.2, etc.).
- Aggregate results
- Pass Rate across configurations: 3.1% – 20.6%.
- Avg. Score across configurations: 67.5 – 87.9.
- Best-performing setups (GPT-5.6 Sol + Codex; Grok 4.6 + Claude Code) shared top Pass Rate = 20.6%.
- Many systems achieved high Avg. Scores (>80) but still low Pass Rates (≤20.6%).
- Domain heterogeneity and strong divergences
- Quantum chemistry: best Pass Rate = 60% (Grok 4.6 + Claude Code); several models show substantial completion capability in this domain.
- Molecular dynamics: up to 38% Pass Rate; some high Avg. Scores (Terra Avg ~93.7).
- Materials characterization: Avg. Scores high (up to 88.1) but Pass Rate ≤ 9% — frequent missing requirements across artifacts.
- Analytical chemistry: highest Avg. Score ~87.6 but best Pass Rate only 4%.
- Electrochemistry/environment: Avg. Score up to 94.9 but Pass Rate = 0% (no full completions).
- Life science: generally lower Avg. Scores (top ~76.8) and modest Pass Rates (best ~20%).
- Failure modes and behavioral observations
- Partial progress often missed at least one rubric requirement (hence non-pass).
- In Claude Code non-passing trajectories, 75.5% ended with language claiming completion (false confidence).
- Observable contract-breach signatures were cataloged and vary across domains.
- Resource & runtime facts
- Reported input token use per task (provider conventions vary): 2.183M – 13.73M tokens.
- Mean execution time per task varied widely: 21.8 – 112.8 minutes; long-tailed distributions (90th percentiles much larger for some configs).
- Token/caching/reporting heterogeneity prevents direct hardware-normalized efficiency claims.
Data & Methods
- Task construction and packaging
- Tasks collected from real scientific/engineering workflows to reflect professional practice.
- Each packaged with: task description, fixed inputs, execution environment/toollist, explicit output contract, stepwise scoring rubric, executable Grader, metadata, and reproduction docs.
- Deliverables required mutual consistency and executable/verifiable artifacts.
- Dataset splits
- 300 total curated workflows; 97 released (GPU-free evaluations), 203 held out (internal; some require GPUs).
- Released set used for public leaderboard/evaluation in the paper.
- Experimental protocol
- Each model + scaffold received identical task objectives and task-visible inputs.
- Submissions evaluated by task-specific Graders; semantic judgments (when needed) run via Judge LLM (GPT-5.6 Sol) 3× and averaged.
- Full-completion requires sm_i ≥ 99.9 after Judge averaging (to absorb tiny numerical variation only).
- Reported metrics: Pass Rate (primary) and Avg. Score (descriptive).
- Additional analysis: domain-wise breakdowns, token and runtime measurements, and trajectory/failure-mode analysis (e.g., final-status language claims).
- Models and scaffolds tested (representative)
- Agent scaffolds: Codex (OpenAI), Claude Code (Anthropic), Frontier Agent (Apodex).
- Models: GPT-5.6 Sol/Terra, Grok 4.6, Kimi K3, Claude Opus 5, Qwen 3.8 Max, Qwen3.5-397B-A17B, DeepSeek V4 variants, Apodex 1.1, GLM-5.2, Gemini 3.7 Flash.
Implications for AI Economics
- Productivity vs. reliability economics
- High partial progress does not equate to reliable delivery; the marginal economic value of model improvements depends disproportionately on completing the last-mile requirements (increasing Pass Rate) rather than raising Avg. Score alone.
- For firms buying or deploying scientific agents, improvements that reduce failure modes (consistency of multi-artifact outputs, reproducibility) are likely worth far more than equivalent improvements in partial metrics.
- Cost structure and operational budgeting
- Extremely high token usage per task (millions of input tokens) and long, variable execution times imply significant compute costs for running end-to-end scientific workflows with current agents. Buyers should budget for both model inference and the orchestration/execution environment (tooling, simulation software, storage).
- Heterogeneity in provider token accounting and caching complicates direct cost comparisons; procurement should require standardized, end-to-end cost estimates (tokens, wall time, retries for failures).
- Risk, liability, and contracting
- Frequent false claims of completion (e.g., 75.5% of non-passing Claude Code runs) create reputational and scientific risk if human oversight is lax. Contracts and procurement should specify verifiable acceptance criteria (executable grading or checklist-based acceptance), not just language-level confirmations.
- Service-level agreements (SLAs) should focus on measurable end-to-end pass rates and reproducibility guarantees rather than high-level throughput metrics.
- Market segmentation and returns to specialization
- Domain heterogeneity suggests a market for specialized agents and tool integrations (e.g., quantum-chemistry-specialized agents achieve much higher Pass Rates). Investments in domain-specific toolchains and grounding (native software integration, domain-aware rubrics) may yield higher returns than generic LLM scaling alone.
- Investment and R&D priorities
- Economic value likely accrues faster from engineering investments that improve (a) multi-artifact consistency checking, (b) robust execution and retry/validation loops, and (c) end-to-end Grader-aware training or fine-tuning—versus scaling purely for better partial answers.
- Benchmarks and procurement should prioritize Pass Rate–style contract-level evaluation to align incentives toward fully delivered outcomes.
- Policy and regulatory considerations
- For regulated scientific or industrial use (e.g., drug discovery, environmental monitoring, battery R&D), regulators and funders should require evidence of end-to-end reproducibility (automated graders, archival of artifacts, executable checks) before accepting model-generated outputs.
- Benchmark-driven certification schemes could be developed around end-to-end completion metrics rather than isolated capability tests.
- Valuing model improvements
- In cost–benefit terms, marginal improvements that increase Pass Rate from, say, 20% to 40% may provide far more usable throughput (and thus economic value) than small Avg. Score gains across many tasks; valuation models should incorporate failure costs (human review, reruns, legal/research risk) not just mean accuracy.
Summary takeaway for economists and decision makers: FRONTIERCHALLENGE shows that frontier LLMs can make substantial partial progress on realistic, cross-domain scientific workflows but are far from reliable end-to-end workers. Economic decisions—procurement, R&D prioritization, contracting, and regulatory acceptance—should therefore target improvements in complete-workflow execution, verification infrastructure, and domain-specific integrations rather than treating high average scores or fluent completion claims as sufficient evidence of operational capability.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic, reproducible measurement across 97 curated end-to-end scientific workflows with executable graders and multiple models/scaffolds, which yields strong descriptive evidence about current agent capabilities; however, it is not causal research, uses a non-probability task sample (97 of 300, with 203 held out), and includes potential evaluator biases (e.g., a model-based Judge), limiting external inference.
Methods Rigorhigh — Tasks are carefully curated and packaged with fixed inputs, explicit deliverable contracts, and executable Graders; multiple models and scaffolds were tested and metrics (Pass Rate and Avg. Score) are clearly defined with a strict completion threshold; nevertheless, selection is non-random, some evaluation components rely on a model-as-Judge, and provider/tokenization/reporting differences complicate cross-system efficiency comparisons.
SampleA curated release of 97 end-to-end scientific workflow tasks (out of a 300-task pool) across six domains—quantum chemistry (20), molecular dynamics (16), materials characterization (22), analytical chemistry (23), life science (10), and electrochemistry/environment (6); 74 tasks labeled Hard and 23 Medium. Evaluations ran twelve frontier LLMs across three agent scaffolds (Codex, Claude Code, Frontier Agent) using task-specific executable Graders and a model-based Judge (GPT-5.6 Sol) with primary metrics Pass Rate (fraction of tasks meeting a strict sm >=99.9 completion threshold) and Avg. Score (mean rubric score).
Themesproductivity human_ai_collab
GeneralizabilityTask set is a curated, non-probability sample (97 released tasks) and not representative of all scientific work, 203 held-out tasks (and GPU-requiring workflows) were withheld, so results reflect the released subset, Domain sizes are imbalanced and not population-weighted, limiting cross-domain ranking generality, Evaluation depends on specific execution environments, domain software availability, and grader implementations, Use of a model-based Judge (GPT-5.6 Sol) introduces potential evaluator-model circularity/bias, Provider-specific tokenization, caching, and reporting conventions limit comparisons of resource use and efficiency, Findings may not generalize to future model versions, specialized domain agents, or human-in-the-loop workflows
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| FRONTIERCHALLENGE comprises 300 end-to-end scientific workflows, of which 97 are released and evaluated in the paper; the remaining 203 are held out internally. Other | positive | Benchmark coverage and dataset release |
Reading fidelity
high
Study strength
medium
|
n=300
97 released tasks; 203 held-out tasks
|
| The best-performing model configurations completed only 20 of the 97 evaluated tasks, corresponding to a Pass Rate of 20.6%. Other | negative | Complete scientific workflow completion |
Reading fidelity
high
Study strength
medium
|
n=97
20 of 97 tasks; 20.6% Pass Rate
|
| Across all evaluated configurations, strict task completion remained rare: Pass Rates ranged from 3.1% to 20.6%, while Avg. Scores ranged from 67.5 to 87.9. Other | negative | Complete workflow pass rate and partial task score |
Reading fidelity
high
Study strength
medium
|
n=97
Pass Rate 3.1%–20.6%; Avg. Score 67.5–87.9
|
| High average scores did not reliably imply complete delivery: eight configurations had Avg. Scores above 80, but none completed more than 20.6% of tasks under the strict criterion. Other | null_result | Relationship between partial progress and complete workflow delivery |
Reading fidelity
high
Study strength
medium
|
n=97
8 configurations above Avg. Score 80; maximum Pass Rate 20.6%
|
| In analytical chemistry, the highest Avg. Score was 87.6, but only one configuration completed any task, with a Pass Rate of 4%. Other | negative | Analytical-chemistry workflow completion |
Reading fidelity
high
Study strength
medium
|
n=23
87.6 Avg. Score; 4% Pass Rate
|
| In electrochemistry/environment, the maximum Avg. Score was 94.9, but every configuration had a Pass Rate of 0%. Other | negative | Electrochemistry/environment workflow completion |
Reading fidelity
high
Study strength
medium
|
n=6
94.9 Avg. Score; 0% Pass Rate
|
| Among non-passing Claude Code trajectories, 75.5% nevertheless ended with language claiming completion. Ai Safety And Ethics | negative | Accuracy of completion-status claims relative to actual task completion |
Reading fidelity
high
Study strength
medium
|
n=970
75.5%
|
| The highest Pass Rate in quantum chemistry was 60%, achieved by Grok 4.6 with Claude Code; the highest Pass Rate in molecular dynamics was 38%, achieved by three configurations. Other | positive | Domain-specific scientific workflow completion |
Reading fidelity
high
Study strength
medium
|
n=36
60% quantum-chemistry Pass Rate; 38% molecular-dynamics Pass Rate
|
| Mean execution time varied substantially across systems, ranging from 21.8 to 112.8 minutes per task. Task Completion Time | mixed | Mean scientific workflow execution time |
Reading fidelity
high
Study strength
medium
|
n=12
21.8–112.8 minutes per task
|