A simple, exact metric reveals humans use implicit conventions to avoid mistakes far more than AI partners do: human pairs outperform literal-hint predictions by about 26 percentage points, AI pairs show near-zero advantage, and human-AI teams fall in between—implying that compatibility with human conventions, not self-play rank, predicts teaming success.

Main Finding

The paper introduces the "convention gap"—the difference between the failure probability predicted from literal communication alone and the observed failure rate—as an objective metric of implicit (conventional) communication in cooperative AI evaluation. Applied to Hanabi play logs (≈101K plays), the metric shows a clear gradient of implicit-communication use: human-human pairs have a large positive gap (+26.2 percentage points), AI-AI pairs are near-calibrated (−0.7 pp), and human-AI pairs fall in between (+16.4 pp). The gap is concentrated on plays where no literal hint exists (untouched cards), varies by AI partner, correlates inversely with partner loss, and is reproducible on controlled Off-Belief Learning agents.

Key Points

• Definition: Convention Gap = mean posterior probability of a life loss computed from literal hint information only − observed life-loss rate. Positive gap → players succeed more than literal information predicts (evidence of implicit conventions).
• Exact posterior: In Hanabi, because of a finite deck and deterministic hint rules, the posterior over card identities (and hence the failure probability for a play) is exactly computable by enumeration—no learning or sampling needed.
• Summary results (≈100,956 plays across datasets):
  • Human-human (hanab.live): gap = +26.2 pp.
  • AI-AI (HOAD): gap = −0.7 pp (nearly calibrated to literal info).
  • Human-AI (HanabiData): gap = +16.4 pp.
• Concentration on zero-/one-hint plays: the human gap is largest when the played card received no hints (≈+46 pp), and collapses near zero once two or more hints touch the card.
• Variation by partner: within human-AI games the literal-information posterior available to humans was similar across AI partners (predicted failure ≈38–41%), but observed human failure rates ranged 14.4%–34.4% and convention gaps varied +6.2 to +24.1 pp; the AI that elicited the largest gap caused the fewest human failures.
• Validation: Off-Belief Learning (OBL) agents with controlled convention content produced the expected hierarchy of gaps (+1.6 pp with no conventions up to +21.7 pp as conventions are reintroduced), supporting the metric's behavioral interpretation.
• Controls and robustness: calibration checks with rule-based AIs, binning by posterior, bootstrap CIs and clustering, and analysis by hint count support the interpretation that the gap isolates non-literal (conventional) information.

Data & Methods

• Task & environment: Standard 2-player Hanabi ("No Variant"), where hints are limited to naming color or rank and pointing to matching cards; misplays cost shared life tokens.
• Posterior computation: exact weighted enumeration over feasible card identities given the literal information set ht (hints on the card, visible hands, fireworks, discards, tokens). Steps:
• Apply hard filters from hints to each card’s candidate identities.
• Weight surviving candidates by remaining unseen copies.
• Sum weights for playable vs. unplayable candidates to get per-card failure probability.
• (Optional fast per-card approx omits joint-hand conditioning).
• Full posterior conditions jointly on the whole hand by enumerating consistent joint assignments with falling-factorial weights; marginalize to the played card.
• Datasets and scale:
  • hanab.live: 425 completed human 2-player games → 9,017 play records (bot accounts filtered).
  • HOAD: 100 games × 49 AI pairings (7×7) → 62,890 plays.
  • HanabiData (human-AI): 2,040 games from 240 players paired with 3 AI types → 29,049 plays (15,472 human plays).
  • Total ≈100,956 play actions.
• Failure-rate metrics:
  • Player Loss: fraction of an agent’s own plays that fail.
  • Partner Loss: fraction of partner plays that fail when paired with this agent.
• Hint-quality diagnostics recorded per hint:
  • Disambiguation power = fraction of candidate identities eliminated by the hint.
  • Playability rate = fraction of hinted cards that are currently playable.
• Validation & robustness: bootstrapping, agent-level clustering, binning by posterior, comparison to OBL agent hierarchy, and checks that AI behavior is calibrated to the posterior (so deviations reflect human implicit channels).

Implications for AI Economics

• Evaluation and procurement:
  • Standard AI-AI benchmarks (self-play/cross-play rankings) can misrepresent an agent’s value in human teams. The convention gap is a candidate metric for procurement and evaluation when the AI will operate with humans—organizations should measure convention compatibility, not just self-play score.
• Product-market fit and adoption:
  • Compatibility (how well an AI elicits and conforms to human conventions) is a distinct product attribute. Buyers will value agents with higher compatibility with their human workforce or user base even if those agents do not top AI-AI leaderboards. This can drive market segmentation—agents optimized for human teaming vs. agents optimized for self-play performance.
• Network effects and complementarities:
  • Human-AI teaming exhibits complementarities: an AI’s marginal value depends on partner type and conventions. The convention gap quantifies one dimension of complementarity and can be used to price contracts, staffing decisions, or match agents to teams.
• Investment and R&D incentives:
  • Investors and R&D managers should consider funding human-centered coordination research (e.g., zero-shot coordination, OBL-style grounding) because convention compatibility affects downstream utility. Metrics like the convention gap enable return-on-investment estimates for human-compatible features.
• Productivity, risk, and externalities:
  • Mis-evaluating an AI’s human compatibility (relying on AI-AI benchmarks) risks deployment failures, reduced productivity, or increased error costs in collaborative tasks. Regulators and firms should account for these coordination externalities when assessing safety, certification, or insurance.
• Standards, certification, and market signaling:
  • A measurable, comparable statistic (like the convention gap) supports certification regimes and labels for "human-compatible" cooperative AIs. Such signaling can reduce search and matching costs in procurement and increase trust.
• Limitations & scalability:
  • The convention gap is exactly computable here because Hanabi has a finite known state and deterministic hint rules. Extending the metric to richer real-world settings will require approximations (e.g., learned literal-baseline models, causal identification of literal vs. conventional signals) and careful validation. Economic models and policy should account for estimation error when generalizing beyond structured games.
• Practical uses for economists and decision-makers:
  • Incorporate convention-gap-like measures into cost–benefit analyses of AI deployment for collaborative tasks.
  • Use the metric in comparative evaluations when choosing AI partners for human teams, and in designing incentive/compensation schemes that reflect complementarities.
  • Factor convention compatibility into pricing/licensing and product-differentiation strategies (agents optimized for human compatibility can command premiums or serve niche markets).

Overall, the convention gap offers a practical, interpretable diagnostic separating literal-information performance from convention-enabled human coordination—an attribute with direct implications for evaluation, procurement, market structure, and policy when AI systems are deployed to collaborate with humans.