General-purpose LLMs can help scale reproducibility checks but cannot substitute experts: AI-assisted teams matched human-only teams on reproduction rates, yet humans caught significantly more major errors, and autonomous AI achieved just a 37% success rate.
Verifying results of published social sciences research is essential but expensive, costing hundreds of dollars per study. With AI tools like ChatGPT becoming widespread, we tested whether they could help scientists check if research findings can be reproduced. We assigned 288 researchers to 103 teams working with no AI, with AI as an assistant, or AI leading the work with minimal human input. Human teams and AI-assisted teams performed similarly on most tasks, but humans caught more critical errors. AI working autonomously achieved a 37% reproduction rate, making it potentially useful for automated screening when human review is cost-prohibitive. These results nonetheless show that human expertise remains essential for reliable scientific validation. Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.
Summary
Main Finding
Human-only teams and human teams assisted by a general-purpose LLM (ChatGPT) achieved similarly high computational reproducibility (94% vs. 91%), but human-only teams found significantly more major coding errors. Fully autonomous, low-oversight AI-led teams performed much worse (37% reproduction rate), detected fewer errors, proposed weaker robustness checks, and took longer. Conclusion: current general-purpose LLMs can scale low-cost screening but do not replace expert human judgment for reliable empirical verification; hybrid human–AI systems show no measurable advantage over humans alone on most outcomes and may reduce detection of major errors.
Key Points
- Experimental design: randomized controlled trial with 288 researchers organized into 103 teams (teams of ~3) across 7 events (Feb–Nov 2024).
- Treatment arms: human-only (no AI), AI-assisted (humans use ChatGPT freely), AI-led (ChatGPT does most work; humans provide minimal oversight and are restricted from directly inspecting article/code/data).
- Primary outcomes: computational reproducibility (success rate, time), error detection (coding errors by severity), and proposed/implemented robustness checks.
- Reproduction rates: human-only 94%, AI-assisted 91% (comparable), AI-led 37% (substantially lower).
- Error detection: human-only teams detected significantly more major coding errors than AI-assisted teams; AI-led teams detected far fewer errors across all categories.
- Robustness checks: AI-led teams proposed weaker checks; AI-assisted and human-only teams were similar on most robustness outcomes.
- Time: AI-led teams required more time to complete tasks than human-led teams.
- AI tooling: teams used ChatGPT (GPT-4/GPT-4o variants during the study), with file uploads, a Python execution environment, and internet access. AI-assisted/AI-led teams received mandatory one-hour ChatGPT training; human-only teams were prohibited from using any AI.
- Data collection: teams had 7 hours per task; materials provided included article PDFs, online appendices, original replication packages, and screenshots of target exhibits. AI chat transcripts were collected for AI arms.
- Task difficulty: papers included coding errors (some previously identified by authors but not publicly disclosed); studies came from leading social science journals; tasks were nontrivial and representative of high-skill reproducibility work.
- Interpretation caveat: the AI-led condition represents a lower bound for autonomous LLM capability (models and agentic approaches evolve rapidly). Results reflect the specific models, configurations, and protocols used in 2024.
Data & Methods
- Participants: 288 researchers (mix of master’s/PhD students, postdocs, faculty, and nonacademic PhD holders) randomized into 103 teams of 3. Randomization stratified by preferred software (R vs Stata) and participation mode (in-person vs virtual).
- Interventions:
- Human-only: no AI allowed.
- AI-assisted: human teams could use ChatGPT (file uploads, code execution, etc.) without restriction.
- AI-led: teams relied primarily on ChatGPT; humans could not directly read the article/code/data (they uploaded these to ChatGPT and interacted through the model).
- Materials: 12 target studies across events; each event included two studies with pre-identified coding errors (one R, one Stata). Teams received article PDFs, replication packages, and screenshots to reproduce specific exhibits.
- Tasks & timeline: 7 hours total to (1) reproduce pre-specified results, (2) detect coding errors, (3) propose/implement up to two robustness checks. Teams reported completion via templated time logs and submitted lists of identified errors and robustness checks.
- AI setup: paid ChatGPT access with models able to process files and run Python; model versions varied across events (GPT-4 family and successors available during Feb–Nov 2024). AI-assisted/led teams underwent standardized training; chat logs were archived and analyzed (including prompt-text similarity).
- Outcomes measured: reproduction success (binary), time to reproduce, counts and severity of coding errors detected, content/strength of robustness checks, and qualitative assessment of team workflows. Statistical comparisons across randomized arms were used to infer treatment effects.
- Incentives & limitations: participants were offered coauthorship (no monetary pay); no performance-based incentives. This reduces strategic manipulation but may affect effort. AI-led teams were constrained by protocol to approximate an agentic LLM with limited human oversight.
Implications for AI Economics
- Labor substitution vs. complementarity
- Short run: general-purpose LLMs (as configured here) are not full substitutes for expert reproducibility labor—human expertise remains essential for high-quality verification. AI assistance did not raise overall reproducibility rates above human-only levels and reduced detection of major errors, suggesting complementarity is fragile and may produce underperformance on important quality dimensions.
- Screening and scale: the 37% autonomous reproduction rate from AI-led teams indicates potential for low-cost, large-scale automated screening to prioritize human review when budgets are constrained. This creates a two-tier workflow: cheap AI pre-screening followed by expert adjudication for flagged or borderline cases.
- Productivity and cost trade-offs
- AI-led workflows may lower per-study costs but at the expense of missed critical errors and longer processing times in this study. Decision-makers (journals, funders, institutions) should weigh cost savings against quality risks and consider mixed-process designs to optimize expected value (e.g., AI triage + human audit).
- Quality assurance and market structure
- Demand for specialized verification expertise likely persists and may even grow (audit, adjudication, AI prompt engineering, domain-specific model fine-tuning). This can sustain or increase the premium for high-skilled reproducibility workers.
- New service markets may emerge: scalable AI-driven screening platforms, human-in-the-loop verification services, and domain-trained reproducibility agents. Buyers will trade off price, speed, and error-detection probability.
- Risks to human capital and incentives
- Overreliance on AI assistance may erode human diagnostic skills over time (expertise erosion), potentially increasing future verification costs and reliance on AI systems. Policy and organizational design should preserve human oversight and training to mitigate skill loss.
- Policy and institutional implications
- Journals and funders should be cautious about delegating verification to general-purpose LLMs without robust human oversight. AI can be incorporated for initial triage, but standards and audits are needed to ensure critical errors are not missed.
- Investment in domain-specific models, toolchains, and standardized reproducibility protocols (including datasets and well-documented code) can improve the effectiveness of AI support and reduce error rates.
- Research and development priorities
- Economic research should evaluate the cost-effectiveness of hybrid verification pipelines, model fine-tuning for reproducibility tasks, incentive designs to maintain reviewer effort, and long-run labor-market effects of increased AI adoption in high-skill scientific tasks.
- Experimental work on prompt engineering, agentic architectures, and task-specific training data is crucial to close the gap between assisted and purely human performance.
Overall, the study implies that while LLMs can expand capacity for screening reproducibility at scale, current general-purpose models are insufficient to replace expert human verifiers without material quality losses. For policy and market design, hybrid systems that preserve human judgment for critical assessments are the prudent path while investing in targeted AI improvements.
Assessment
Claims (12)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Verifying results of published social sciences research is expensive, costing hundreds of dollars per study. Organizational Efficiency | negative | cost_per_reproducibility |
Reading fidelity
high
Study strength
medium
|
hundreds of dollars per study
|
| We randomly assigned 288 researchers to 103 teams working under three conditions (human-only, AI-assisted, AI-led). Other | null_result | experimental_assignment |
Reading fidelity
high
Study strength
high
|
n=288
|
| Human-only teams achieved a 94% reproduction rate when attempting to reproduce published quantitative social science results. Research Productivity | positive | reproduction_rate |
Reading fidelity
high
Study strength
high
|
94%
|
| AI-assisted teams (using ChatGPT as a collaborative tool) achieved a 91% reproduction rate. Research Productivity | positive | reproduction_rate |
Reading fidelity
high
Study strength
high
|
91%
|
| Human-only and AI-assisted teams performed similarly on most outcomes. Research Productivity | null_result | multiple reproduction-related outcomes (overall performance parity) |
Reading fidelity
high
Study strength
medium
|
comparable performance
|
| Human-only teams identified significantly more major coding errors than AI-assisted teams. Error Rate | positive | major_coding_error_detection_rate |
Reading fidelity
high
Study strength
medium
|
identified significantly more major coding errors
|
| AI-led (autonomous ChatGPT with minimal human oversight) teams achieved only a 37% reproduction rate. Research Productivity | negative | reproduction_rate |
Reading fidelity
high
Study strength
high
|
37% reproduction rate
|
| AI-led teams detected fewer errors across all categories than human or AI-assisted teams. Error Rate | negative | error_detection_rate_all_categories |
Reading fidelity
high
Study strength
medium
|
detected fewer errors across all categories
|
| AI-led teams proposed weaker robustness checks than human or AI-assisted teams. Research Productivity | negative | robustness_check_strength |
Reading fidelity
medium
Study strength
medium
|
proposed weaker robustness checks
|
| AI-led teams required more time to complete the reproduction tasks than human or AI-assisted teams. Task Completion Time | negative | time_to_complete_tasks |
Reading fidelity
medium
Study strength
medium
|
required more time
|
| Although AI working autonomously achieved a 37% reproduction rate, it could be useful for automated screening when human review is cost-prohibitive. Research Productivity | mixed | potential_value_for_screening |
Reading fidelity
high
Study strength
speculative
|
37% autonomous reproduction rate
|
| Overall, LLM assistance did not produce measurable advantages for human-supervised verification and was associated with reduced detection of major errors, meaning expert human judgment remains indispensable for reliable empirical verification. Research Productivity | negative | effect_of_AI_assistance_on_verification_quality |
Reading fidelity
high
Study strength
medium
|
no measurable advantages; associated with reduced detection of major errors
|