Agentic research assistants fall short: on AARRI-Bench the best-performing agent configuration succeeds on just 68.3% of granular researcher tasks and frequently overlooks subtle but critical judgments, suggesting current systems cannot fully replace human researchers.
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
AARRI-Bench shows that state-of-the-art agentic systems achieve only partial success at emulating human researchers—best config reached 68.3% and commonly missed subtle, critical judgments—indicating agents are not yet replacements for human researchers.
As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3\% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Our data is released at https://github.com/AARR-bench/AARRI-bench.
Summary
Main Finding
The paper introduces AARRI-Bench (the first benchmark in the AARR "Act As a Real Researcher" series) to evaluate whether frontier LLMs and agent harnesses behave like real researchers. Across 82 manually crafted, research-relevant tasks, the best model+harness combination (Mini-SWE-Agent + Claude Opus 4.7) attains only a 68.3% overall success rate and frequently misses subtle, researcher-relevant details. The authors conclude that improving researcher-like AI will require modeling research behavior and judgment — not just more complex scaffolding.
Key Points
- Benchmark scope and purpose
- AARRI-Bench focuses on “easy-for-humans / error-prone-for-agents” research subtasks that test researcher qualities (integrity, uncertainty awareness, independent judgment, careful verification), not only final execution.
- 82 tasks manually crafted from real researcher pain points; tasks organized along two dimensions:
- Horizontal (task scenarios): Context, Mindset, Hands-on, Interaction.
- Vertical (agent scope): S1 Adaptation (32%), S2 Integration (28%), S3 Innovation (27%), S4 Open-ended (13%).
- Task design and evaluation
- Tasks packaged in the Harbor format (instruction.md, task.toml, environment/, solution/, tests/), executed in isolated cloud environments (Daytona, Modal).
- Two-layer scoring: classic 0/1 final-completion reward (for aggregate comparisons) plus fine-grained unit tests for diagnostic analysis.
- Dataset and tasks are publicly released (GitHub link in paper).
- Experimental coverage
- Evaluated 16 model+harness combinations using state-of-the-art closed-source models (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.3 Codex, Qwen 3.6 Plus) and several open-source models (MiniMax-M2.7, Kimi-K2.6, DeepSeek-V4-Flash), together with harnesses Claude Code, Hermes Agent, and Mini-SWE-Agent.
- Performance highlights
- Best overall: Mini-SWE-Agent + Claude Opus 4.7 → overall 68.3% (by coarse 0/1 metric).
- Per-category for that best combo: Context 64.7%, Mindset 76.9%, Interaction 76.2%, Hands-on 57.1%.
- Simpler harness (Mini-SWE-Agent) with a strong model outperformed or matched more elaborate harnesses, suggesting scaffolding complexity alone is not sufficient.
- Central qualitative finding
- Agents still miss subtle but critical research judgments (e.g., recognizing dead-ends, applying methodological conservatism, handling uncertainty), indicating a gap in research behavior modeling.
Data & Methods
- Task construction
- Three-stage human-in-the-loop creation: free task proposals from researchers, guided expansion with targeted feedback, consolidation and de-duplication into final 82 tasks.
- Tasks are manual (not synthetic), selected to reflect real research workflows and common agent failure modes.
- Taxonomy
- Horizontal: Context, Mindset, Hands-on, Interaction.
- Vertical: S1–S4 representing increasing autonomy and intellectual contribution.
- Evaluation pipeline
- Harbor framework for reproducible, containerized task execution.
- Environments built via Docker; runs executed in cloud recommended by Harbor (Daytona, Modal).
- Models and harnesses
- Closed-source: Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.3 Codex, Qwen 3.6 Plus.
- Open-source: MiniMax-M2.7, Kimi-K2.6, DeepSeek-V4-Flash.
- Harnesses: Claude Code, Hermes Agent, Mini-SWE-Agent (open-source).
- Metrics
- Coarse-grained final completion (0/1) used for aggregate comparisons; fine-grained unit tests used for diagnostics and ablation.
- Availability
- All tasks and benchmark artifacts are released (link provided in paper).
Implications for AI Economics
- On labor substitution and complementarities
- The benchmark shows substantial but incomplete automation: agents can handle many entry-to-mid-level research tasks (success rates up to ≈68%) but miss critical judgment-sensitive items. This implies partial substitution: automation will likely compress demand for routine, execution-focused research tasks while increasing demand for human skills that emphasize judgment, interpretation, ethical oversight, and open-ended creativity.
- Firms will face shifting complementarities: value will accrue to workers who can supervise, validate, and augment agent outputs (i.e., human-in-the-loop roles), and to teams that can integrate agents into workflows safely.
- Productivity measurement and quality-adjusted output
- Standard productivity metrics that count completed tasks will overstate effective research output if they ignore quality dimensions (reproducibility, methodological rigor). Economic evaluations should use quality-adjusted output metrics; AARR-style benchmarks help quantify the gap between task completion and researcher-quality performance.
- Incentives for tool builders and procurement
- The result that simpler harnesses + stronger models can outperform more complex scaffolding suggests firms and vendors may prefer investing in model capability and targeted behavior modeling over ever-more elaborate orchestration infrastructure. However, robustness and safety remain crucial; procurement should weigh model accuracy against research-quality failure modes.
- Labor-market and wage dynamics
- Demand for mid-level research assistants (interns, research engineers doing repetitive code/experiment tasks) may decline or be redefined; wages may bifurcate—falling for routinized tasks, rising for high-skill oversight/interpretation roles. Training and re-skilling investments will be important.
- Organizational risk and governance
- Deployment of agents with non-negligible error rates on judgment-sensitive tasks creates reputational, regulatory, and scientific-reproducibility risks. Firms doing R&D should require quality assurance processes (human audit, probabilistic calibration, unit-testing regimes) and incorporate benchmarks like AARRI to set acceptance thresholds.
- Policy and market design
- Public funding agencies and journals may need to adapt evaluation standards (e.g., require human attestations of methodological checks or benchmarked AI-assisted methods) to prevent degradation of scientific standards and to internalize externalities arising from poorly validated agent outputs.
- Research investment implications
- The benchmark indicates research effort should be directed toward: (a) modeling research behavior (uncertainty, stopping rules, meta-reasoning), (b) tools for calibrated confidence and provenance, and (c) evaluation frameworks that capture researcher-like quality. These are promising areas for technology investment with high social return, as they directly affect the reliability of automated R&D.
- Market for verification services
- Given agents’ proclivity to miss subtle errors, a market will likely expand for verification, auditing, and validation services (both human and automated) — analogous to quality-control sectors in manufacturing — which could create new high-value job categories.
Overall, AARRI-Bench quantifies an important middle ground: frontier agents are increasingly useful and productive for research workflows but are not yet substitutes for human researchers where judgment, ethics, and subtle methodological scrutiny matter. Economic analyses, procurement decisions, and policy design should therefore focus on quality-adjusted productivity, human–AI complementarities, and governance mechanisms.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports systematic empirical evaluations on a novel benchmark and provides quantitative performance (e.g., 68.3% best-case success). These results credibly demonstrate current capability limits, but they do not establish causal claims about broader impacts, and findings depend on benchmark design, model selection, and evaluation criteria that could bias results.
Methods Rigormedium — The authors run extensive experiments across multiple frontier models and agentic scaffolds and release the dataset, which supports reproducibility; however, assessment depends on the benchmark's task construction, scoring rubrics, potential subjectivity in grading nuanced researcher behaviors, and limited detail on inter-rater reliability, task sampling, and robustness checks.
SampleAARRI-Bench (AARRI-Bench/AARRI-Bench) — a curated set of granular research-oriented tasks designed to probe professionalism, thoroughness, field sensitivity, ethics, and nuanced reasoning; evaluated across multiple frontier models and agentic systems (examples include Mini-SWE-Agent with Claude Opus 4.7 and other leading LLM-based agents); quantitative success rates reported (best configuration 68.3%); dataset and evaluation code released on GitHub.
Themeshuman_ai_collab productivity
GeneralizabilityBenchmark tasks and scoring may not cover the full diversity of real-world research disciplines or workflows, Results depend on the particular agent scaffolds, model versions, and prompts tested and may change as models update, Evaluation of nuanced research judgment likely involves subjective judgments and may vary by rater or domain expertise, Lab-bench performance may not translate directly to field productivity, collaboration dynamics, or institutional constraints, Ethical, cultural, and domain-specific norms in real-world research may not be fully captured
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Developer Productivity | positive | proficiency on complex, long-horizon coding tasks and autonomous experiment execution |
Reading fidelity
high
Study strength
low
|
not reported
|
| Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment, and consequently remain unable to fully replace human researchers. Research Productivity | negative | ability to match human researchers on field sensitivity, research ethics, and nuanced scientific judgment |
Reading fidelity
high
Study strength
low
|
not reported
|
| We conceptualize the AARR (Act As a Real Researcher) benchmark series to evaluate whether agents can emulate the professionalism, thoroughness, and nuanced reasoning of human researchers in granular research scenarios. Research Productivity | positive | ability to emulate professionalism, thoroughness, and nuanced reasoning |
Reading fidelity
high
Study strength
high
|
not reported
|
| We propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in the AARR series that focuses on granular research scenarios rather than macro-level execution capabilities. Adoption Rate | positive | existence and focus of the AARRI-Bench benchmark |
Reading fidelity
high
Study strength
high
|
not reported
|
| Extensive experiments across frontier models and agentic systems reveal that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only a 68.3% success rate on AARRI-Bench. Task Completion Time | negative | success rate on AARRI-Bench tasks |
Reading fidelity
high
Study strength
medium
|
68.3% success rate
|
| Agents frequently overlook subtle yet critical details that are obvious to real human researchers. Output Quality | negative | frequency of overlooking subtle, critical research details |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Research Productivity | positive | effectiveness of approaches (behavioral modeling vs. scaffolding) for producing researcher-like AI |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| The AARRI-Bench dataset and code are released at https://github.com/AARR-bench/AARRI-bench. Other | positive | availability of dataset and code |
Reading fidelity
high
Study strength
high
|
not reported
|