Evidence (896 claims)
Search and filter individual claims pulled from the papers. Looking for a specific finding ("what's the effect on wages?"), you're in the right place. Want to compare whole outcome categories against each other instead? Use the Evidence Explorer.
The board below groups claims two ways: by broad theme (nine paper-level topics) and by outcome category (the 34 claim-level outcomes that the Explorer and Syntheses also use).
Browse by theme
Nine broad, paper-level topics. Click one to filter the claims below.
Adoption
9875 claims
Filter claims →
Productivity
8807 claims
Filter claims →
Governance
7870 claims
Filter claims →
Human-AI Collaboration
7560 claims
Filter claims →
Org Design
4892 claims
Filter claims →
Innovation
4781 claims
Filter claims →
Labor Markets
4004 claims
Filter claims →
Skills & Training
3308 claims
Filter claims →
Inequality
2332 claims
Filter claims →
Claims by outcome category
Counts by direction of finding. These are the same 34 outcome categories the Explorer compares and the Syntheses are written for. A linked row has a published synthesis.
| Outcome | Positive | Negative | Mixed | Null | Total |
|---|---|---|---|---|---|
| Other | 870 | 233 | 116 | 1066 | 2363 |
| Governance & Regulation | 976 | 451 | 218 | 133 | 1809 |
| Organizational Efficiency | 949 | 224 | 144 | 88 | 1416 |
| Technology Adoption Rate | 764 | 287 | 141 | 122 | 1325 |
| Research Productivity | 501 | 152 | 74 | 362 | 1101 |
| Output Quality | 542 | 216 | 69 | 69 | 896 |
| Decision Quality | 387 | 198 | 94 | 54 | 740 |
| Firm Productivity | 513 | 67 | 101 | 27 | 714 |
| AI Safety & Ethics | 249 | 303 | 73 | 36 | 667 |
| Market Structure | 190 | 192 | 134 | 27 | 548 |
| Task Allocation | 243 | 77 | 91 | 36 | 452 |
| Innovation Output | 291 | 33 | 55 | 20 | 401 |
| Skill Acquisition | 206 | 72 | 65 | 21 | 364 |
| Employment Level | 133 | 63 | 115 | 22 | 335 |
| Fiscal & Macroeconomic | 153 | 79 | 52 | 32 | 323 |
| Task Completion Time | 206 | 37 | 12 | 15 | 272 |
| Firm Revenue | 179 | 52 | 29 | 5 | 266 |
| Consumer Welfare | 130 | 76 | 47 | 13 | 266 |
| Inequality Measures | 48 | 137 | 51 | 6 | 242 |
| Worker Satisfaction | 101 | 81 | 25 | 13 | 220 |
| Error Rate | 84 | 110 | 11 | 5 | 210 |
| Wages & Compensation | 98 | 47 | 30 | 10 | 185 |
| Regulatory Compliance | 88 | 73 | 17 | 7 | 185 |
| Automation Exposure | 66 | 64 | 33 | 16 | 182 |
| Team Performance | 105 | 29 | 30 | 11 | 176 |
| Training Effectiveness | 109 | 22 | 14 | 21 | 168 |
| Developer Productivity | 114 | 21 | 14 | 8 | 158 |
| Job Displacement | 12 | 90 | 24 | 1 | 127 |
| Hiring & Recruitment | 57 | 9 | 9 | 5 | 80 |
| Skill Obsolescence | 6 | 56 | 9 | 1 | 72 |
| Social Protection | 43 | 17 | 8 | 2 | 70 |
| Creative Output | 35 | 21 | 9 | 4 | 70 |
| Labor Share of Income | 18 | 21 | 17 | 1 | 57 |
| Worker Turnover | 15 | 16 | — | 4 | 35 |
| Industry | — | — | — | 1 | 1 |
Essay quality changes little while students have AI access but improves in style and relevance one week later when students write unaided.
Open-ended essay assessments (higher-order skills) collected immediately (with AI access for treatment group) and one week later (unaided) in the randomized experiment; quality measured on dimensions including style and relevance.
Hybrid (human-AI) performance, analyzed at the individual forecaster level, is trimodal: most people either deferred to the model (matching it) or rubber-stamped a prior guess (performing worse than the model alone), while a minority engaged in genuine complementary reasoning and reached accuracy matching or even exceeding the market.
Pilot empirical analysis comparing individual forecasters' hybrid forecasts to both the model and the Polymarket benchmark; claims reported at individual level in the paper.
In established open-source projects, adopting an AI coding agent makes code modestly more complex but does not crowd out the human newcomers that a project depends on.
Synthesis of the paper's DiD results: no significant decline in newcomer inflow, unchanged onboarding/retention, correlational beginner-task measure unchanged, and measured modest increases in complexity metrics.
Code detected as likely to be generated by LLMs shows substantial intra-repository code clones.
Code-clone analysis applied to code flagged by LLM-detection tools within the same repositories (detector-based proxy approach).
Projected yield distributions vary substantially across locations, with some lower productivity sites exhibiting yield increases under future climate scenarios.
Results from simulated climate-projection experiments across multiple locations showing heterogenous yield distribution changes, including increases in some lower-productivity sites.
Bias transfer from the LLM is asymmetric: agency is suppressed in female-target essays while male-target writing remains largely unaffected.
Comparative analysis within the participant data showing differential effects by target gender (female-target vs male-target essays) in the N = 123 study; reported asymmetry in the paper summary.
Specifying instructions for AI-agents does not necessarily lead to better results.
Project-level before/after comparison of performance metrics for projects that created instruction files (pre/post comparison across 148 projects using the 15,549 PRs).
Analysis of recent benchmark evidence including SWE-bench Verified, EvoClaw, and LangChain's multi-agent coordination studies demonstrates both the transformative potential of the agentic paradigm and its current limitations.
Empirical/benchmark analysis referencing SWE-bench Verified, EvoClaw, and LangChain multi-agent studies as sources of evidence; the paper analyzes these benchmarks qualitatively or comparatively (specific sample sizes and quantitative effect sizes not stated in the abstract).
An explicit thinking mode raises rank-order correlation without moving accuracy.
Empirical comparison of reasoning modes showing increased rank-order correlation (e.g., Spearman/Fisher-z) when explicit 'thinking' mode is used, with no significant change in accuracy.
Overall, complementarity is attainable in multi-agent regression but obstructed in classification under natural conditions on local aggregation and loss functions.
Synthesis of the paper's proved positive results for regression and negative impossibility results for classification within the tree-based HAI framework (theoretical proofs; no empirical sample).
In regression under squared loss, complementarity is equivalent to Euclidean distance minimization from the ground-truth vector.
Analytic equivalence proved in the paper for the tree-based model under squared loss (mathematical derivation; no empirical sample).
The autonomously generated manuscripts also diverged in length, details, and quality.
Reported qualitative comparison of the LLM-assisted manuscripts produced by each agent indicating differences in length, level of detail, and overall quality between the two agents' outputs.
No single LLM dominates across engine types, highlighting the importance of specific tasks and tradeoffs between speed and accuracy.
Empirical observation from cross-engine evaluations reported in the paper; descriptive conclusion without numeric dominance metrics or sample sizes in the excerpt.
The evaluations implemented by the initiative demonstrate that AI enabled modeling tools perform better at discussion and basic qualitative tasks than with causal reasoning and quantitative error fixing.
Result reported from the implemented evaluations comparing relative performance across task categories (discussion/qualitative vs causal reasoning/quantitative error fixing); no quantitative effect sizes or sample sizes provided in the excerpt.
When engines from the sd ai project are coupled with different LLMs, their performance on these evaluations reveals variability across different AI tools.
Empirical statement in the paper based on applying the implemented evaluations to different engine+LLM combinations; no numeric performance metrics or sample sizes reported in the excerpt.
These findings demonstrate the feasibility and current limits of automated expertise mapping.
Synthesis/conclusion based on model performance (e.g., MAE results) and observed limitations reported across evaluations.
Models with near-identical overall strength show qualitatively different capability profiles.
Observed differences in capability-profile axes for models with similar aggregate scores in the tournament.
For molecular sonification, the gain is representational rather than predictive.
Reported outcome for molecular structure to music task indicating improvements in representation/sonification quality but not in predictive performance.
Some merged PRs introduce new lint or security findings while simultaneously removing existing issues (i.e., merges sometimes involve both addition and removal of issues).
Before-and-after static analysis (Pylint and Bandit) of merged PRs showing coexistence of introduced and removed findings in observed diffs.
Cross-model validation reveals architecture-level trade-offs independent of specific LLMs: Dual Process excels at numeric/temporal queries (65-90% accuracy) while RAG excels at historical retrieval (60-85% accuracy).
Empirical cross-model tests across six LLMs; reported accuracy ranges for different query types and architectures.
Clarifying-question prompts produced mean rubric scores of 6.67 out of 8, higher than raw prompts but lower than checklist-improved prompts.
Reported mean rubric scores in the abstract showing clarifying-question prompts scored 6.67, compared to 5.67 for raw and 7.50 for checklist.
LLMs often generate responses with the structural clarity associated with early-career engineers, yet they display persistent weaknesses in factual grounding and contextual interpretation.
Qualitative and comparative analysis of LLM responses against the expert rubric during the audit (six commercial LLMs); observed patterns in response form and substantive content.
Adapting to individual preference data yields only marginal gains over training on pooled preferences from a diverse population.
Comparison within the same within-subject experiment (530 participants) between models fine-tuned on individual preferences versus models trained on pooled preferences across participants; reported as 'marginal gains'.
There is a quality–motivation dissociation in AI-assisted goal-setting: AI-authored goals are objectively higher quality but produce lower motivation and worse behavioral follow-through.
Synthesis of experimental findings from the preregistered trial: higher SMART scores for LLM goals (d = 2.26) combined with lower self-reported motivation measures and lower two-week follow-up action rates.
Fine-tuning and reinforcement learning improve in-distribution performance, but generalization to unseen part families remains limited.
Experiments reported in the paper/abstract applying fine-tuning and reinforcement learning to models evaluated on BenchCAD; observed improvements on in-distribution data and limited generalization to unseen families.
Across 10+ frontier models, current systems often recover coarse outer geometry but fail to produce faithful parametric CAD programs.
Empirical evaluation reported in the paper/abstract across more than ten contemporary multimodal / large language models on the BenchCAD dataset; observed pattern that coarse outer geometry is often recovered while faithful parametric program synthesis fails.
Larger models do not consistently outperform smaller ones on tool-use tasks.
Empirical observations from the paper's evaluations across the five function-calling benchmarks.
Aesthetic and functional attributes load onto a single latent factor, suggesting users perceive quality as a unified construct rather than separable aesthetic and functional dimensions.
Factor analysis (or similar latent-variable analysis) on participant ratings of multiple attributes showing a single dominant factor combining aesthetic and functional attributes.
Across 78 endpoints serving 12 model families, the same model on different endpoints differs in mean accuracy by up to 12.5 points on math and code.
Empirical measurement across 78 endpoints and 12 model families comparing mean accuracy on math and code tasks.
Fluent users' failures occur alongside greater success on complex tasks.
Combined analysis of task complexity, success outcomes, and failure incidence in the 27K transcripts showing that fluent users both attempt and have greater success on complex tasks even while experiencing more failures.
LLMs are able to extract signals from unstructured text (financial news headlines) but have limitations without explicit quantitative optimization.
Interpretation in discussion/conclusion: empirical finding that LLM-based portfolios beat naive diversification but underperform AI-optimized strategies, implying LLMs extract signals from text yet lack full optimization capability.
Whether LLM-based assistants improve or degrade code quality remains unresolved: existing studies report contradictory outcomes contingent on context and evaluation criteria.
Review finds mixed/contradictory findings across included studies regarding code quality effects.
The system tends to be factually correct when it answers but often omits information (i.e., 'the system is right when it answers — it just leaves things out').
Interpretation combining reported factual accuracy (85.5%) with low completeness (0.40) from benchmark results.
ASC (adaptive stopping criterion) halts harmful refinement but incurs a 3.8 pp confidence-elicitation cost.
Reported experiment with ASC showing that it prevents harmful iterative refinement yet causes a measured cost described as 3.8 percentage points due to confidence elicitation.
Only o3-mini (+3.4 pp, EIR = 0%), Claude Opus 4.6 (+0.6 pp, EIR ~ 0.2%), and o4-mini (+/-0 pp) remain non-degrading under self-correction; GPT-5 degrades by -1.8 pp.
Reported measured changes in accuracy (percentage-point changes) and measured EIR values for the named models after applying iterative self-correction across the experiment suite.
Across 7 models and 3 datasets (GSM8K, MATH, StrategyQA), we find a sharp near-zero EIR threshold (<= 0.5%) separating beneficial from harmful self-correction.
Empirical experiments reported across 7 LLMs and 3 benchmark datasets (GSM8K, MATH, StrategyQA) comparing outcomes of iterative self-correction as a function of measured EIR.
Removing safety layers made the system less useful: structured validation feedback guided the model to correct outcomes in fewer turns, while the unconstrained system hallucinated success.
Qualitative and quantitative comparisons from the deployed evaluation across the three conditions (observations about turn counts, validation-feedback loops, and model hallucinations in unconstrained condition over the 25 scenario trials).
Practitioners negotiate model performance via technical and political means.
Observational data from the ethnography showing technical adjustments, benchmarks, and political negotiation (e.g., with regulators or management) to establish acceptable performance.
These effects are observed across a variety of tasks, including mathematical reasoning and reading comprehension.
Trials included multiple task types (explicitly naming mathematical reasoning and reading comprehension); cross-task analysis reported.
Providing issue-specific design guidance reduces design violations, but substantial non-compliance remains.
Intervention experiments in paper: agents were given issue-specific design guidance and resulting patch compliance measured; reported reduction in violations but remaining non-compliance.
Models performed well on commonly discussed topics but struggled with specialized health data.
Task-level performance comparison across topics in the elicited population statistics: better accuracy on commonly discussed topics, poorer performance on specialized health data tasks.
In a preliminary experiment, giving models web search access degraded predictions for already-accurate models, while modestly improving predictions for weaker ones.
A preliminary comparative test where some models were given web search access and changes in predictive performance were observed: degradation for already-accurate models and modest improvement for weaker models.
The top four models are statistically indistinguishable (mean score 0.147–0.153) while a clear tier gap separates them from the remaining four models (mean score <= 0.113).
Reported mean performance scores across 8 models and statement of statistical indistinguishability for the top four vs lower-tier four; numerical means provided.
Testing revealed AI excels at computational tasks but consistently misses nuanced factors like new construction rent premiums and infrastructure proximity impacts, validating the framework's hybrid structure as essential for professional-grade underwriting.
Findings from the controlled ChatGPT-4 test on the single 150-unit scenario: qualitative and comparative observations showing AI handled computations well but failed to capture specific local-market nuances, leading authors to endorse a hybrid human-AI framework.
AI assistance in safety engineering is fundamentally a collaboration design problem rather than merely a software procurement decision: the same tool can either degrade or improve analysis quality depending entirely on how it is used.
Synthesis of the formal framework and analytic results in the paper (theoretical argument; no empirical sample reported).
Extensive synthetic experiments show that policy regularizations reshape the narrative on what is the best DRL method for inventory management.
Paper states results from extensive synthetic experiments that change which DRL methods are considered best under policy regularization; abstract does not provide the experimental sample size, specific methods, or quantitative comparisons.
PPS gains are task-dependent: gains are large in high-ambiguity business analysis tasks but reverse in low-ambiguity travel planning tasks.
Task-level analysis across the three domains (business, technical, travel) within the controlled study (60 tasks total); authors report differential performance patterns by domain/ambiguity.
We evaluate 14 LLMs under zero-shot prompting and retrieval-augmented settings and witness a clear performance gap.
Experimental evaluation reported in the paper: authors state they ran experiments on 14 different large language models, under zero-shot and retrieval-augmented configurations, and observed differing performance across models.
Convergence after exemplar exposure occurred by both tightening of estimates within a measure family and by agents switching measure families.
Agent-level tracking across stages showed two patterns following exemplar exposure: (1) reduced within-family dispersion (tighter estimates) and (2) categorical switches in measure selection by some agents, as recorded across the 150-agent sample.
Choice of scaffold materially affects outcomes: an open-source scaffold outperformed vendor-provided scaffolds by up to approximately 5 percentage points.
Comparative experiments across three scaffolding approaches (vendor scaffolds and at least one open-source scaffold) showing up to ~5 percentage point differences in measured outcomes.