How researchers pay participants shapes what we learn about human–AI teamwork: inconsistent or misaligned incentives bias measured effort, trust and accuracy, so the authors propose a practical Incentive‑Tuning Framework to calibrate and transparently report pay schemes across studies.

Main Finding

Monetary incentives used in crowdsourced empirical human–AI decision-making studies are heterogeneous, under-documented, and typically designed heuristically (mostly flat base pay ± bonuses). This inconsistency threatens internal and external validity of findings. The authors synthesize prevailing practices via a reflexive thematic analysis of 97 papers and propose the Incentive‑Tuning Framework (plus a reporting template and public repository) to standardize how researchers design, tune, and report incentive schemes.

Key Points

• Dataset and scope

  • 97 in‑scope papers (crowdsourced, human‑subject, empirical studies of human–AI decision‑making; literature through Nov 2023 plus a precompiled list).
  • Excluded non‑crowdwork, non‑decision tasks, and purely formative/qualitative studies.

• Main thematic findings

  • Components: Nearly all papers (88/97) mention incentive components. The dominant distinction is base pay vs. bonus.
    • Base pay reported in 83/97 papers; often a flat amount or hourly rate.
    • 47/88 papers reported both base pay and performance bonus; 41/88 only base pay; 5/88 only bonus.
  • Design practice: Most pay rates/amounts are chosen heuristically, rarely justified; a few reference platform norms or minimum wage.
  • Documentation: Many studies give sparse or no rationale for incentive choices; some omit incentive descriptions altogether.
  • Impact: Authors note (and prior work suggests) that incentive design can alter participant motivation, engagement, strategy, and thus measured trust, reliance, accuracy, or other outcomes.
  • Ethics & fairness: Fair compensation is sometimes acknowledged but not systematically operationalized.

• Outputs from the paper

  • Incentive‑Tuning Framework: practical guidelines for designing, reflecting on, and documenting monetary incentive schemes in human–AI decision studies.
  • Reporting template and public GitHub repository to share incentive schemes and rationales for published studies.

Data & Methods

• Inclusion criteria: empirical, evaluative human‑AI decision‑making tasks run on crowdsourcing marketplaces with monetary incentives for crowd workers.
• Search strategy: started from an 81‑paper list covering to 2021, then searched major venues (ACM IUI, CHI, FAccT, CSCW, AIES, HCOMP) for 2021–Nov 2023 using keywords like “human‑AI collaboration” and “human‑AI decision‑making.” Initial screening yielded 86 candidates; final set was 97 papers after full screening.
• Extraction: for each paper extracted study goals, domain, stakes, participant/platform details, task role, incentive descriptions and any discussion of participant motivation.
• Analysis: reflexive thematic analysis (inductive, semantic, critical) following Braun & Clarke’s six phases; coding and theme development focused on explicit descriptions of incentives and discussions of motivation; authors documented positionality and used a public codebook and checklist.
• Reproducibility: dataset of excerpts, codebook, checklist, and other supplementary materials are publicly available in a GitHub repo.

Implications for AI Economics

• Incentives as a treatment variable: Incentive design is endogenous to participant behavior. For AI economics experiments (and papers measuring human responses to ML systems), incentives should be treated explicitly as experimental treatments or covariates, not background protocol.
• External validity and generalizability: Flat monetary pay on crowdsourcing platforms may not reproduce real‑world decision stakes (financial, legal, reputational). Misaligned incentives can bias estimates of trust, reliance, bias correction, or welfare effects of AI assistance.
• Measurement and identification concerns:
  • Selection effects: pay level influences who participates (skill, effort, opportunity cost), changing sample composition and estimated effects.
  • Moral hazard and gaming: performance‑contingent bonuses can induce strategic behavior that confounds the mechanism of interest (e.g., overfitting to bonus metric).
  • Incentive interaction with AI properties: incentive structures may interact with model accuracy, explainability, and other treatments, producing heterogeneous treatment effects.
• Cost–benefit and labor considerations: standardizing reporting of pay and rationale enables better cost accounting for large‑scale experiments and facilitates ethical benchmarking of crowd worker compensation in economic analyses.
• Recommendations for practitioners and researchers in AI economics
  • Report incentives fully: include base pay, bonuses, pay rate, estimated task time, how amounts were chosen (pilot, platform norms, minimum wage), eligibility rules, and observed payouts.
  • Pre‑specify and pre‑register incentive schemes where possible; treat incentives as part of the experimental design (and report them in methods).
  • Align incentives with study goals: choose performance metrics used for bonuses that reflect the real‑world objective you aim to emulate, but consider distortions and gaming risks.
  • Pilot and calibrate: estimate task time and behavioral response via pilots to set compensation that avoids perverse incentives or disengagement.
  • Run robustness checks: vary incentive levels or structures (flat vs. performance‑contingent) as sensitivity analyses to assess how conclusions depend on incentive design.
  • Use the authors’ Incentive‑Tuning Framework and reporting template; deposit incentive schemes and rationales in shared repositories to enable meta‑analysis and replication.

Overall, economic analysis of human–AI systems must internalize incentive design as a central experimental and normative variable: it shapes observed behavior, affects cost and fairness assessments, and should be standardized and transparently reported to advance cumulative knowledge.