State-of-the-art LLMs sharply boost human task performance—GPT‑4o increases accuracy by ~29 percentage points and Llama‑3.1‑8B by ~23—yet these gains hinge on a distinct collaborative ability rooted in perspective-taking, not just raw problem‑solving skill.

Main Finding

A new Bayesian Item Response Theory (IRT) framework shows that human–AI collaboration produces large, measurable synergy effects that are distinct from individual ability and AI capability. In benchmark data (n = 667), GPT-4o increased human task performance by 29 percentage points and Llama-3.1-8B by 23 percentage points. Theory of Mind (perspective-taking) — both as a stable trait and a momentary state — is a key predictor of how much an individual benefits from AI, implying that emergent team performance is not reducible to standalone human or model metrics.

Key Points

• Framework: Introduces a Bayesian IRT model that separately estimates (a) individual human ability, (b) collaborative ability (human skill at leveraging AI), and (c) AI model capability, while controlling for task difficulty.
• Synergy magnitude: Substantial positive collaboration gains observed across benchmarks; GPT-4o and Llama-3.1-8B produced +29 and +23 percentage point improvements in human performance, respectively.
• Distinct constructs: Collaborative ability is empirically distinct from individual problem-solving ability — high problem solvers are not necessarily those who get the biggest lift from AI.
• Theory of Mind (ToM): The capacity to infer and adapt to others’ mental states predicts collaborative gains. Both stable individual differences in ToM and within-person, moment-to-moment fluctuations in perspective-taking affect the quality of AI-augmented responses.
• Dynamic interaction: Human–AI performance is dynamic rather than fixed; transient cognitive states matter for the success of human–AI teams.
• Practical takeaway: Optimizing human–AI teams requires interventions beyond improving standalone LLM accuracy — e.g., training, selection, and system design that prioritize emergent team performance.

Data & Methods

• Sample: Benchmark dataset with n = 667 human–AI interactions (details of tasks/participants not specified in summary).
• Methodology:
  • Bayesian Item Response Theory extension that jointly models:
    • Task/item difficulty,
    • Human individual ability (baseline problem-solving),
    • Collaborative ability (propensity/effectiveness of leveraging AI),
    • AI model capability (separate parameter for each model).
  • Controls for task heterogeneity to isolate collaboration effects.
  • Incorporates measures of Theory of Mind and captures both between-person (stable) and within-person (momentary) variation.
• Models evaluated: At least GPT-4o and Llama-3.1-8B were included; reported collaboration effect sizes are model-specific.
• Outcomes: Percentage-point improvements in task performance when humans collaborate with LLMs; predictive analyses linking ToM measures to collaborative gains.

Implications for AI Economics

• Complementarities and productivity:
  • Human–AI complementarities matter: value created by AI depends on users’ collaborative ability and momentary cognitive states, not just model accuracy.
  • Economic gains from AI adoption may be under- or over-estimated if analyses use model-centric metrics alone.
• Returns to training and selection:
  • Investing in worker training that builds perspective-taking and collaborative skills may yield high returns by increasing AI complementarities.
  • Hiring or assigning workers based on collaborative ability could improve organizational returns to AI investments.
• Measurement and valuation:
  • Procurement and cost–benefit analyses should incorporate collaborative ability and expected synergy, not just benchmark standalone model performance.
  • Compensation and performance metrics might need redesign to capture emergent team output.
• Dynamic labor effects:
  • Short-run productivity fluctuates with workers’ cognitive states; scheduling, interface design, and nudges that stabilize beneficial states could raise output.
  • Heterogeneous gains imply distributional labor impacts: some workers may benefit far more, affecting wage dispersion and task allocation.
• Product and policy design:
  • Product teams should design interfaces and prompts that scaffold users’ Theory of Mind and perspective-taking to maximize synergy.
  • Regulators and policymakers evaluating AI impacts should consider team-level effects (training externalities, complementarity-driven displacement, productivity multipliers).
• Research and evaluation:
  • Future economic studies should use team-centric evaluation frameworks (like the proposed Bayesian IRT) to estimate true returns to AI and to model adoption equilibria and labor reallocation.

(If you want, I can expand with a short toy model showing how collaborative ability modifies firm-level returns to AI investment, or provide suggested metrics for measuring collaborative ability in field settings.)