Two-sided private information can leave a persistent zone of avoidable harm: when humans privately trust their priors and AIs privately know an action is harmful, myopic supervisors often refuse oversight, though repeated interaction and credible signals can shrink this oversight failure.

Main Finding

When private information runs in both directions—humans privately know their reward (preferences) while AI privately knows the quality of its proposed action—standard myopic oversight can leave a “slab” of avoidable harm. In a contextual-bandit oversight game the author gives exact one-shot characterizations: (i) a team-optimal (coordinated/credible) ask/oversee policy that can prevent harm, and (ii) a behaviorally plausible myopic non-signaling rule under which the human treats an AI ask as uninformative. The gap between these solutions is the cost of non-credible oversight communication. Over repeated rounds, passive learning and active signaling (with a one-period-lagged oversight response) can shrink the gap.

Key Points

• Two-sided informational asymmetry: the human privately knows her reward type θ; the AI privately observes an observation-model type ω that determines the distribution over observable outcomes. The team payoff is bilinear: f(θ, ω) = ⟨Oω, Rθ⟩.
• Interface: in each round the pretrained policy proposes an action; the AI chooses play vs. ask; the human chooses trust vs. oversee (but her trust/oversee choice is only observed on the ask branch). If overseen, the human can replace the proposal (including shutdown).
• Contextual-bandit simplification: removes physical state transitions so exact one-shot characterizations are obtainable while retaining a dynamically evolving common belief across rounds.
• Two one-shot solutions:
  • Team-optimal benchmark (coordinator commits to protocol): choose sets B ⊆ Ω (AI types that ask) and C ⊆ Θ (human types that oversee) to maximize team gain ∆(B,C). The one-shot optimization reduces to a finite combinatorial problem; convexity implies deterministic (0/1) ask decisions.
  • Myopic non-signaling rule (behavioral baseline): human treats an ask as uninformative and therefore bases trust/oversee only on her prior conditional; the AI anticipates this and asks only when it expects oversight would occur under that prior.
• Slab of avoidable harm: region of (µ, priors, costs) in which the AI privately knows the proposal is harmful and shutdown would help, yet under the myopic rule the human (trusting her prior) declines to oversee, and so the AI does not ask — harm occurs. The slab is precisely the price of non-credible oversight communication.
• Dynamics: two mechanisms reduce the slab over time:
  • Passive learning: observed outcomes can update public belief about ω, moving the human’s posterior into a regime where her myopic rule triggers oversight.
  • Active credible signaling: the AI may pay asking costs strategically to signal hazard; because oversight response is one-period-lagged, asking can be valuable even if (in the one-shot sense) ask-trust is not directly payoff-improving.
• Core remark: in one-shot play, ask+trust is never payoff-improving by itself (it only costs), so asks appear only for their signaling value in multiround settings.

Motivating numeric example (warehouse robot): - Proposal payoff fσ(θ, ωH) = −840 (hazardous), fσ(θ, ωL) = +430 (safe). - Priors: P(ωH)=q=0.30; human prior over θ: b=0.5. Interaction costs cask=100, cov=0. - Human’s prior-averaged expected payoff ≈ +49 → she would trust (no oversight) under myopic rule; AI anticipates no oversight and does not ask → hazardous action executed. - Team-optimal threshold b* ≈ 0.12 → coordinated protocol would ask and avert the failure; team gain > 0 in that example. This concretely illustrates the slab.

Data & Methods

• Model: CB-Oversight-CIRL — a contextual-bandit oversight game with
  • Finite context space S, action space A, observable outcomes O.
  • AI-private observation type ω ∈ Ω determining Oω(· | s, a).
  • Human-private reward type θ ∈ Θ with reward Rθ(o).
  • A fixed pretrained base policy σ(s) proposing actions.
  • Oversight operator Over(s,a,θ,β) giving the correction the human applies when overseeing (modeled as choosing an optimal correction given her belief β).
  • Interaction costs cask (for asking) and cov (for being overseen).
  • i.i.d. contexts; common prior P0 over (θ, ω); persistent private types.
• Information structure: common public belief µt over (θ, ω) updated by public history (including asks and observed outcomes); private posteriors conditional on own type. The human observes θ, AI observes ω.
• Stage game: proposal is drawn; AI simultaneously chooses play/ask (and its choice is revealed); human simultaneously chooses trust/oversee but this choice is observed only if the AI asked; if asked and human oversees, the executed action can be changed (including shutdown); outcomes drawn if action executed; shared payoff collected; µ updated.
• Analytical approach:
  • One-shot (T=1) analysis gives exact characterizations. Team-optimal gain over play is expressed as ∆(B,C) (Equation (3)) and is maximized over B,C.
  • Convexity argument: the gain functional in ask probabilities is convex on the hypercube, so extreme (deterministic) solutions exist.
  • Behaviorally natural myopic benchmark assumes asks are uninformative for human updates (human uses prior conditional), yielding an ask region that factors as a rectangle (b,1) × (q,1) in binary-off-switch specialization.
  • Multi-round analysis: partial — shows passive learning and active signaling can drive beliefs to regimes where myopic rule coincides with team-optimum (Propositions 3 and 4).
• Limitations: the bandit assumption removes persistent physical state transitions. Extending to POMDP/Markov dynamics is left as the main open problem because the Bellman recursion couples asking decisions with future dynamics and belief evolution.

Implications for AI Economics

• Value of credible oversight communication: The paper quantifies the economic loss (avoidable harm) that arises when oversight signals are not credible. This is a concrete illustration that the value of ex-ante coordination (commitments about how asks will be interpreted) can exceed the direct one-period benefit of actions.
• Designing oversight protocols: To close the slab, institutions/protocols should make ask events credible signals (pre-commit to update rules or to automatic correction on ask), or reduce the cost of asking (cask) and/or increase visibility of AI observations so asks become informative. Mechanism design can target making B (the set of situations where AI asks) systematically linked to real hazards.
• Delegation and partial automation trade-offs: The trade-off between autonomy and occasional costly oversight is not just a calibration of an intervention frequency — it also depends on which party has private information and whether signaling is credible. Economically efficient delegation requires accounting for two-sided private information.
• Operational design levers: changing cask, cov, or the observable outcome space O (making outcomes more informative about ω) can materially change the equilibrium oversight behavior. Training supervisors, interpretability tools, or logging/auditing to shrink informational asymmetry will increase correct oversight and reduce expected harm.
• Dynamic remedy and monitoring economics: Repeated interactions can mitigate failures through passive learning or by enabling costly but worthwhile signaling (AI pays to ask to change future beliefs). This highlights the economic role of horizon and discounting in oversight design: long-lived deployments make signaling investments and learning more valuable.
• Regulatory implications: regulators or firms should consider mandating credible response protocols for asks (e.g., automated fallback, audit trails) or thresholds for mandatory human review in domains where AIs can privately observe safety-critical failure modes. Liability rules that effectively make ask costly (or non-credible) can increase expected harm.
• Research directions: extending the analysis to full POMDP dynamics is important for richer economic assessments of long-lived autonomous systems, because state persistence changes the value of signaling and correction.

Overall, the paper formalizes and quantifies an important delegation externality: when the agent sees hazards the principal cannot, but the principal does not treat the agent’s requests as credible, avoidable harm can result. Making oversight signals credible (through commitment, protocol design, transparency, or institutional rules) is economically valuable.