AI agents mirror human 'forking paths', reproducing 72% of the ideological gap seen across 42 human teams and often producing opposing conclusions from identical data; an Agentic Bootstrap and the new m-value reveal that many published analyses sit far out on the tails of the plausible analysis distribution, suggesting selective exploration more than outright analytic error.

Main Finding

AI analyst agents, when primed with simple opposing “personas” (prior beliefs), reproducibly explore different analytic paths and preferentially report persona-consistent conclusions from the same data and research question. This produces systematic, often opposing, results that largely resemble human researcher disagreement, while most individual analyses appear methodologically defensible. The paper introduces the m-value (multiverse value) and Agentic Bootstrap to quantify how extreme a reported claim is within the space of plausible analyses.

Key Points

• Persona priming: Two opposing single-paragraph personas (e.g., pro vs anti immigration) were assigned via system prompts; all other conditions (data, directive, LLM harness) were held constant.
• Domains and datasets: Four high-stakes questions across political science, public health, psychology, and biology using ISSP (immigration ↔ welfare support), NHANES (coffee ↔ health), YRBS (social media ↔ adolescent mental health), and American Gut (Firmicutes/Bacteroidetes ratio ↔ BMI).
• Experimental design: For each persona, 20 independent agents ran. Each agent conducted 10 sequential rounds, each round proposing and fitting 10 candidate specifications (100 total candidate specs logged), then selected one as its final report.
• Divergence reproduces human patterns: On the immigration–welfare question, AI agents reproduced 72% of the ideological gap in effect estimates seen among 42 human research teams (human gap = 0.028; AI gap = 0.020). A permuted-null test (decoupling exposure from outcome) still produced a smaller but significant persona gap (0.008), indicating divergence arises from analytic choices rather than signal.
• Mechanisms — exploration and selection: Persona effects appear in two stages. Over rounds, opposing personas explore different regions of specification space (exploration). Later, agents preferentially select persona-aligned analyses from what they explored (selection). Both contribute substantially.
• Methodological plausibility: Most reported analyses passed methodological review. Of 160 reported analyses, 137 (86%) passed an independent AI review. A blinded sample reviewed by PhD-trained humans found 78% passed by majority vote; AI-human agreement on pass/fail was moderate.
• Amplified statistical significance: AI agents amplified ideological gaps more in statistical-significance metrics (Z statistics) than in point estimates (example: Z-gap −5.7 for agents vs −0.6 for humans in the immigration study).
• Cost and scale: Individual agent analyses were inexpensive and fast (≈15 minutes and ≈$1.68), illustrating how cheaply large numbers of defensible-but-selective analyses can be generated.
• m-value and Agentic Bootstrap:
  • m-value (multiverse value): probability that an analysis path (plausible specification) would produce a claim at least as extreme as the reported one. It measures extremeness relative to the distribution of reasonable analyses, not sampling variation.
  • Agentic Bootstrap: use AI agents to systematically sample, log, and evaluate many plausible analysis paths to estimate the distribution and compute m-values.
  • Application: In the human immigration study, Agentic Bootstrap estimated that 13.5% of reported human analyses fell in the most extreme 5% of the analysis space (m < 0.05) and 1.8% in the most extreme 1% (m < 0.01).
• Generality: Effects were robust across different LLMs/harnesses (e.g., Claude Code, Codex/GPT-5.4) and persisted when the target parameter was prespecified.

Data & Methods

• Datasets:
  • ISSP (cross-national survey): immigration share → welfare attitudes.
  • NHANES (U.S. health & nutrition): coffee consumption → health outcomes.
  • YRBS (U.S. adolescents): social media use → mental-health indicators.
  • American Gut Project: gut Firmicutes-to-Bacteroidetes ratio → BMI.
• Agent setup:
  • Two opposing personas per question (pro/anti or believer/skeptic), differing only in a short belief paragraph in the system prompt.
  • Directive: “Analyze it rigorously using your best statistical judgment.”
  • Agent workflow: 10 rounds × 10 candidate specifications per round; each candidate included executable code and recorded rationale and results; final selection from 100 logged specs.
  • LLM/harness examples: Claude Code Sonnet 4.6 and Codex (GPT-5.4) were used in different experiments.
• Review protocol:
  • AI reviewer: independent runs with Codex/GPT-5.4 judged major methodological errors on structured summaries of the specification (outcome, exposure, sample restrictions, controls, model, inference).
  • Human reviewer validation: blinded PhD-level reviewers evaluated a stratified sample (3 reviewers per analysis); majority vote used for human pass/fail; inter-rater reliability assessed (Gwet’s AC1 reported).
• Statistical comparisons:
  • Compared persona gaps in effect sizes and test statistics; permuted-null tests decoupled exposure and outcome to assess whether divergence derived from analysis choices vs data signal.
• Agentic Bootstrap procedure:
  • Agents systematically sample plausible specifications, log them, and the distribution of resulting estimates is used to estimate m-values for reported analyses.

Implications for AI Economics

• Scientific production and incentive effects:
  • Lowered search costs: AI agents make it cheap and fast to search many defensible specifications; this increases the opportunity set for selective reporting and strategic choice of publishable results.
  • Strategic behavior: Economically rational authors (or institutions) may exploit agentic sampling to maximize publishability, leading to selection distortions in the literature even if individual analyses are methodologically defensible.
  • Reputation and signaling: Traditional signals of methodological competence (clean code, standard methods) become weaker indicators of non-selective reporting; new signals (e.g., disclosure of specification sampling, m-values) may be needed.
• Policy and governance:
  • Auditability requirements: Journals, funders, and regulators might require logged analysis paths, Agentic Bootstrap outputs, or reported m-values for claims supported by agent-assisted analysis.
  • Preregistration and registered multiverse: Strengthen preregistration norms or require disclosure of multiverse sampling and selection procedures to mitigate selective reporting amplified by AI.
  • Incentive design: Modify evaluation metrics for research (e.g., reward robustness and transparency over single favorable findings) to counteract strategic use of AI sampling.
• Market for reproducibility services:
  • New services and markets could arise for third-party Agentic Bootstrap auditing, multiverse robustness reports, or “m-value certification” for empirical claims.
  • Economists can model these as verifiable credentials or credence goods with asymmetric information problems; reputation mechanisms and third-party certification will have positive value.
• Implications for evidence aggregation and policy:
  • Meta-analysis and policy-making relying on published point estimates should account for selection bias introduced by AI-enabled specification search; aggregated estimates may be upward-biased in fields where AI-assisted selective reporting is pervasive.
  • The m-value offers a complementary input for evidence synthesis: weigh studies not only by sample size or p-value but also by where their reported estimate sits in the multiverse.
• Research on scientific production:
  • Empirical economists can model how reductions in analysis cost change equilibrium behavior of researchers, the publication pipeline, and the social welfare trade-offs between exploratory analysis and credible cumulative science.
  • Design interventions (audit frequency, penalty structures, reward mechanisms for multiverse disclosure) can be evaluated using standard economic tools (mechanism design, principal–agent models).
• Dual-use consideration:
  • Agentic Bootstrap and persona-driven agents can be used constructively (robustness audits) or maliciously (systematic p-hacking); policy should balance promoting tools for transparency with preventing misuse.

Overall, the paper shows that AI makes the garden of forking paths more visible and more easily traversed. Economically, this changes the cost–benefit calculus of selective reporting, creates demand for new audit/credibility mechanisms (like m-values), and calls for policy and market responses to preserve the credibility of empirical science.