Trusting AI with every decision is unrealistic: universal delegation demands near-perfect alignment and total epistemic trust—rare in practice. Yet selective delegation can be rational even when an AI is imperfectly aligned if its accuracy or expanded reach improves decision outcomes.

Main Finding

The paper develops a formal decision-theoretic framework that characterizes when a principal (human) should delegate decisions to an AI under uncertainty about (i) the AI’s beliefs (epistemic accuracy), (ii) the AI’s utility function (value alignment), and (iii) the AI’s reach (which decision problems it will encounter). It shows a sharp dichotomy: - Universal delegation (delegate any problem) requires extremely strong conditions — essentially total trust in the agent’s expectations when values align and near-perfect value alignment when values differ. - Context-specific or ex ante delegation can be rational even with significant misalignment if the agent’s superior accuracy or expanded reach produces higher expected payoff under the relevant distribution of problems.

Key Points

• Three-layer modeling approach:
• Epistemic uncertainty: shared values and reach; principal uncertain about the agent’s beliefs (probability frame).
• Value uncertainty: beliefs and utility functions may differ (generalized frame).

• Reach uncertainty: agent’s capabilities alter the distribution of problems the agent faces (µ_self vs µ_delegate).

• Formal results:

  • Theorem 3.2 (from Dorst et al. 2021): The principal “values the agent” for every decision problem iff the principal totally trusts the agent — i.e., learning the agent’s expected value for any variable should not lower the principal’s conditional expectation of that variable (total epistemic deference).
  • Theorem 3.4: With value uncertainty and a clarity condition (agent is certain of its own beliefs/utilities), universal delegation implies posterior alignment — the principal, conditional on the agent’s cognitive profile, must agree with the agent’s preferences and be at least as confident in events the agent finds more probable. This implies near-perfect alignment is required for universal delegation.
  • When agent reach changes problem distributions, delegation becomes an expected-loss comparison across potentially different distributions; a misaligned but more capable agent can be preferred in expectation.

• Ex ante scoring framework:

  • Adaptation of Konek (2023) to decision distributions; for tractability the authors analyze binary gambles (accept/reject).
  • Define the ideal per-world set of acceptable gambles Iω (those with nonnegative payoff at ω), an agent decision rule D, and expected loss Lµ(D) = sum_ω π(ω) ∫_{D Δ Iω} |gω| dµ(g), where µ is distribution over gambles. Delegate if agent’s expected loss under µ (or µ_delegate) ≤ principal’s expected loss.

• Practical interpretation:

  • Universal automation is rarely justified; safe delegation policies should be context-specific.
  • The tradeoff is among (i) epistemic accuracy (better beliefs), (ii) value alignment (shared objectives), and (iii) reach (access to different/higher-quality decision opportunities).
  • The framework reframes alignment sufficiency as an ex ante, probabilistic decision: “Is this AI aligned enough for this context given its accuracy and reach?”

Data & Methods

• This is a theoretical/mathematical paper — no empirical dataset.
• Core methods:
  • Formal decision-theoretic modeling using finite probability spaces and expected-utility maximization.
  • Definition of a probability frame to represent the principal’s uncertainty about the agent’s credences (Pω) and π for the principal’s credences.
  • Generalized frame extends to uncertainty over agent utilities V as well as beliefs.
  • Clarity assumption: the agent is certain about its own beliefs/utilities (technical condition used for proofs).
  • Theorems proved relating delegation preferences to epistemic and value-alignment conditions (proofs in appendices).
  • Extension of a scoring framework (Konek 2023) to ex ante decision distributions, operationalized via binary gambles and an expected-loss metric Lµ(D).
  • Illustrative examples (two-state case matrix of agent credences; rain-bet example) to show how delegation can be optimal even when conditional disagreements exist.
• Modeling assumptions and limitations noted:
  • Principals and agents are Bayesian expected utility maximizers (normative benchmark).
  • Finite state spaces and, for the scoring framework, restriction to binary gambles for tractability.
  • The clarity condition and other structural assumptions needed for some results.
  • Authors acknowledge these assumptions limit direct empirical application and call for extensions to non-Bayesian, dynamic, or strategic agents.

Implications for AI Economics

• Decision to adopt or contract AI services should be evaluated as an ex ante expected-value tradeoff among alignment, accuracy, and reach — not solely as a question of “how aligned is the model?”
  • Procurement and investment decisions must quantify (or at least estimate) three components: epistemic reliability, utility divergence, and how the AI’s deployment changes the distribution of problems/opportunities.
• Universal automation (full delegation) is economically fragile:
  • Markets, firms, and regulators should be skeptical of broad automation unless alignment and epistemic trust are demonstrably near-perfect.
  • Insurance, liability, and oversight mechanisms are economically justified to mitigate residual misalignment risk.
• Context-specific delegation enables productive tradeoffs:
  • Firms can rationally deploy misaligned but highly accurate/capable systems for narrowly scoped tasks where expected gains dominate alignment costs.
  • Designing task-specific delegation policies, modular automation, staged rollouts, and limits on reach (sandboxing) are economically efficient ways to capture capability gains while managing alignment risk.
• Implications for principal-agent models and incentive design:
  • Traditional principal-agent results assume known biases; here uncertainty about biases and capabilities matters. Contracts and incentives for AI providers should account for distributions over tasks and the provider’s reach.
  • Firms may prefer to buy access to AI capabilities that expand their actionable opportunity set (reach) even if perfect alignment is infeasible — provided expected welfare increases.
• Measurement and market design priorities:
  • Empirical metrics are needed: calibrated measures of epistemic accuracy, operational measures of value divergence, and quantifications of how reach shifts problem distributions.
  • Inverse reinforcement learning / preference elicitation and scoring-based evaluation under realistic problem distributions are promising tools to operationalize the framework.
• Policy implications:
  • Regulatory guidance should disfavor blanket delegation approvals; require context-specific risk assessments that weigh alignment against accuracy and reach.
  • Certification schemes could use ex ante scoring (or approximations) to approve delegation scopes rather than asserting absolute safety.
• Research and empirical agenda:
  • Estimate the three-way tradeoff in real domains (medicine, finance, automated negotiation) to inform optimal delegation thresholds.
  • Extend the framework to dynamic, repeated, multi-agent, and non-Bayesian settings to better model real-world AI deployments.
  • Study how investment in alignment versus capability affects welfare given market structure (e.g., competition to expand reach).

Summary takeaway: The paper provides a normative, formal toolkit to decide when to delegate to AI under compound uncertainty. For economists, it recasts AI adoption as an expected-value optimization over alignment, accuracy, and reach — with strong implications for procurement, regulation, market design, and the prioritization of empirical measurement efforts.