A pilot using real-money markets finds that pairing people with AI rarely yields uniform gains: most users either copy the model or worsen performance, but a small subset with collaborative traits reasoning with the AI outperforms both the model and typical human forecasts.

Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

Vivienne Ming · July 02, 2026

arxiv correlational medium evidence 8/10 relevance Full text usable extracted full text Source PDF

In a real-money prediction-market pilot, human-AI hybrid forecasting shows a trimodal distribution—most people either defer to the model or rubber-stamp prior beliefs, while a minority with collaborative traits (perspective-taking, intellectual humility, curiosity) achieve complementary gains that match or exceed the market benchmark.

Whether pairing people with AI helps or hurts is usually reported as a single average effect. Using a real-money prediction market (Polymarket) as an objective, externally resolved benchmark, this pilot shows that the value of human-AI collaboration depends on a specific, measurable form of human capital. Analyzed at the level of the individual forecaster, hybrid performance is trimodal: most people either deferred to the model (matching it) or used it to rubber-stamp a prior guess (performing worse than the model alone), while a minority engaged in genuine complementary reasoning and reached accuracy matching or even exceeding (i.e., lower error than) the market itself. Collaborative traits (perspective-taking, intellectual humility, and curiosity) rather than raw cognitive ability or model benchmarks, distinguished who reached that mode. The results are preliminary but statistically robust, and motivate a pre-registered replication now in preparation.

Summary

Main Finding

In a pilot study using real-money Polymarket questions, human–AI hybrid forecasting produced a trimodal distribution of outcomes: most people either (a) deferred to the model (Automators) and matched but did not beat the model, or (b) rubber‑stamped their prior (Validators) and performed worse than unaided humans or the AI; a minority (Cyborgs) engaged in complementary reasoning and matched or exceeded the market benchmark. Crucially, collaborative human capital (perspective‑taking, curiosity, intellectual humility), not raw cognitive benchmarks, predicted who achieved the high-performing Cyborg mode.

Key Points

Design: Pilot preprint; 78 participants in main pool (77 analyzed) organized into 26 three‑person teams; participants produced individual probabilistic forecasts on live Polymarket contracts (30 questions, each team forecast 10).
Outcome metric: Scaled Brier score ×100 (lower = better). Market benchmark = 3.5; four-model AI-only baseline mean = 5.5.
Trimodal performance (participant-level means):
- Human-only: 14.7 (95% CI 14.4–15.0, n = 35).
- Automators: 10.4 (9.7–11.0, n = 24) — better than humans but worse than AI baseline.
- Validators: 31.7 (20.6–42.7, n = 9) — worse than human-only.
- Cyborgs: 3.8 (3.5–4.1, n = 9) — lower error than each individual AI model and matching/exceeding market benchmark.
Statistical evidence: Kruskal–Wallis omnibus χ² across groups p < 10⁻¹³; post-hoc Welch t-tests reported for pairs.
Models (AI-only Brier ×100): GPT-4o & Gemini 3 Pro ≈ 4.6; Qwen3 = 5.5; Llama 3.1 = 7.1.
Human-capital vs. accuracy:
- Among hybrid users, collaborative traits predicted lower error (perspective‑taking r = −0.32, p = .04; curiosity r ≈ −0.26; intellectual humility r ≈ −0.25, trends).
- Raw cognitive ability did not predict hybrid accuracy (g r = −0.10; fluid reasoning r = −0.08; both ns).
- For human‑only forecasters, cognitive and collaborative traits strongly predicted accuracy (intellectual humility r = −0.66, p < .001; g, fluid reasoning, curiosity, perspective‑taking ≈ r = −0.45, p < .01).
Who becomes a Cyborg: Cyborgs scored much higher on collaborative traits vs. other hybrid users (intellectual humility d = 4.1; curiosity d = 4.0; perspective‑taking d = 3.2), with smaller differences in fluid reasoning (d = 1.5) and g (d = 1.7).
Qualitative process: Cyborgs used models for data-bound anchoring while humans explored contingent, ill-posed aspects; Automators outsourced both; Validators used the model to confirm prior beliefs.
Supplementary observations: A Socratic (answer‑withholding) model increased Cyborg rate but reduced user satisfaction; opportunistic EEG in 7 volunteers suggested lower cognitive engagement for Automators.

Data & Methods

Participants: 108 recruited; main study n = 78 (42 students, 36 community adults; mean age 28.7); analyzed n = 77 after one exclusion. Teams of three; individual forecasts collected.
Conditions: Human-only (12 teams), Hybrid with access to one of four LLMs (14 teams). Separate Socratic pool (30 participants; reported separately).
Models tested: Llama 3.1 (8B), Qwen3 (8B), GPT‑4o, Gemini 3 Pro.
Forecasts: Real-money Polymarket contracts resolving Nov 2025–Jan 2026; ground truth o ∈ {0,1}.
Human-capital battery (session 1): Raven’s APM, ICAR (g proxy), IRI perspective‑taking, Comprehensive Intellectual Humility Scale, epistemic curiosity, Brief Resilience Scale, TIPI Big-Five.
Interaction-style taxonomy: Emergent post‑hoc classes — Automators, Validators, Cyborgs (classification collinear with human capital; not experimentally assigned).
Analysis: Participant‑level Brier means, 95% CIs; Kruskal–Wallis omnibus test; Welch’s t-tests for post‑hoc comparisons; Pearson correlations; effect sizes reported as Cohen’s d.
Limitations: Pilot sample; small cell sizes for Cyborgs and Validators (n = 9 each); emergent (non‑randomized) style collinear with human capital → descriptive comparisons only; within‑condition low variance inflates standardized effect sizes; four-model AI baseline limits market comparison; forecasting timestamps relative to market state not fixed. Replication pre-registered and planned.

Implications for AI Economics

Heterogeneous returns to AI: Average treatment effects obscure qualitatively distinct modes of human–AI interaction. Economic estimates of productivity gains from AI should disaggregate interaction styles rather than report single mean effects.
Human capital re‑specification: Standard cognitive benchmarks (g, fluid reasoning) predict solo performance but not who will generate complementarity with AI. Economists and firms should measure and value collaborative traits (perspective‑taking, curiosity, intellectual humility) when assessing complementarity and returns to AI investments.
Labor market and hiring: Comparative advantage in an AI era may shift toward workers with collaborative human capital. Hiring, training, and selection criteria should incorporate measures of collaborative dispositions and the capacity to engage in iterative, complementary reasoning with algorithmic tools.
Task assignment and organizational design: Match tasks and interfaces to likely interaction modes. Design interfaces and workflows that scaffold Cyborg‑style behavior (structured exploration prompts, Socratic scaffolding) to unlock gains; beware of interfaces that make it easy to Automate or Validators that can degrade outcomes.
Policy and evaluation: Benchmarking models (e.g., accuracy metrics) is insufficient for predicting hybrid team performance. Regulators and researchers should use externally resolved, style‑independent outcomes (markets, real decisions) where possible, and consider heterogeneity and distributional effects (some workers may be harmed relative to model-alone performance).
Investment in complementary training: Returns to training programs targeted at collaborative skills may be high relative to investments exclusively boosting cognitive skills or model quality.
Research agenda: Need for pre-registered, adequately powered replications that (i) randomize or manipulate interaction style, (ii) pre-register composite human-capital measures, and (iii) fix forecast timing relative to market state. Also worth exploring incentive designs, interface affordances, and whether similar patterns appear in other economic tasks (decision‑making, diagnosis, coding, policy formation).
Measurement caution: When assessing AI’s economic impact, use disaggregated analyses that identify the fraction of users in each interaction mode; averaging can hide harmful subpopulations (Validators) and underestimate the gains achievable by Cyborgs.

Note: Results are preliminary (pilot) and intended to motivate the planned pre-registered replication.

Assessment

Paper Typecorrelational Evidence Strengthmedium — Uses an objective, externally resolved real-money benchmark and reports statistically robust individual-level patterns, which strengthens internal validity; however, the study is a small pilot without randomized treatment or large representative sample, limiting causal claims and external validity until pre-registered replication. Methods Rigormedium — Strengths include use of an external, incentive-aligned market outcome, individual-level error analysis, and measurement of relevant psychological traits; weaknesses include pilot/sample-size constraints, likely multiple testing and post-hoc mode identification (trimodal clustering) that may be data-driven, limited detail on randomization or experimental control, and potential selection bias of participants. SamplePilot sample of individual forecasters who made predictions on tasks with outcomes resolved in a real-money prediction market (Polymarket); participants completed forecasting both with and without model assistance (hybrid condition) and filled out measures of collaborative traits (perspective-taking, intellectual humility, curiosity) and cognitive ability; exact sample size and recruitment population not specified in the summary but described as a preliminary pilot. Themeshuman_ai_collab skills_training IdentificationCompare forecast errors for humans alone, the AI model alone, and human+AI hybrids using realized outcomes from a real-money prediction market (Polymarket) as an objective benchmark; classify individual-level performance patterns (matching model, rubber-stamping, complementary improvement) and correlate those patterns with measured individual traits (perspective-taking, intellectual humility, curiosity) to infer which human capital predicts complementary gains. No randomized assignment of people to distinct treatment arms is reported; identification therefore relies on within-sample comparisons and associations rather than randomized causal inference. GeneralizabilitySmall pilot sample size limits statistical generalizability, Likely self-selected forecasters or non-representative participants (selection bias), Domain-specific: forecasting tasks on Polymarket may not generalize to other economic tasks (e.g., production, service work, coding), Model type, interface, and task framing unspecified — results may not hold for different AI models or interaction designs, Short-term forecasting tasks with real-money incentives may not translate to longer-run workplace productivity or wage effects, Causal claims limited by lack of randomized assignment and potential confounders

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
The study used a real-money prediction market (Polymarket) as an objective, externally resolved benchmark. Other	null_result	benchmark outcomes (Polymarket market prices / resolution)	Reading fidelity high Study strength high	not reported 0.5
Hybrid (human-AI) performance, analyzed at the individual forecaster level, is trimodal: most people either deferred to the model (matching it) or rubber-stamped a prior guess (performing worse than the model alone), while a minority engaged in genuine complementary reasoning and reached accuracy matching or even exceeding the market. Output Quality	mixed	forecasting accuracy / error relative to model and market	Reading fidelity high Study strength medium	not reported 0.3
Most participants deferred to the model, producing forecasts that matched the model's predictions. Output Quality	null_result	degree of agreement with model predictions / matching rate	Reading fidelity high Study strength medium	not reported 0.3
Many participants used the model to rubber-stamp a prior guess and, as a result, performed worse than the model alone. Output Quality	negative	forecasting accuracy relative to model (worse performance)	Reading fidelity high Study strength medium	not reported 0.3
A minority of participants engaged in genuine complementary reasoning and achieved accuracy matching or exceeding (i.e., lower error than) the Polymarket market itself. Output Quality	positive	forecasting accuracy relative to market (equal or lower error)	Reading fidelity high Study strength medium	not reported 0.3
Collaborative traits — perspective-taking, intellectual humility, and curiosity — rather than raw cognitive ability or model benchmarks, distinguished who reached the complementary (high-performing) mode. Skill Acquisition	positive	engagement in complementary reasoning / attainment of higher forecasting accuracy	Reading fidelity high Study strength medium	not reported 0.3
Raw cognitive ability or model benchmark metrics did not distinguish who engaged in complementary reasoning. Skill Acquisition	null_result	engagement in complementary reasoning (prediction by cognitive ability / model benchmarks)	Reading fidelity high Study strength medium	not reported 0.3
The results are preliminary but statistically robust. Other	null_result	statistical significance / robustness of reported pilot effects	Reading fidelity high Study strength medium	not reported 0.3
These pilot findings motivate a pre-registered replication that is now in preparation. Other	null_result	planned pre-registered replication	Reading fidelity high Study strength speculative	not reported 0.05