Keeping models preference‑free and applying asymmetric costs at decision time typically raises welfare because embedding preferences during training reduces informativeness; but when users cannot compute optimal thresholds, embedding preferences can outperform.

Main Finding

Training predictors to produce preference-free calibrated probabilities (i.e., using strictly proper scoring rules) and applying user-specific preferences ex post (post-processing) weakly dominates embedding preferences into the training loss, provided a diminishing-value-of-information condition holds. Preference embedding typically induces a mean-preserving contraction of the learned posterior distribution (a “capability tax”), which reduces the marginal value of information and so weakly reduces expected welfare for every expected-utility decision problem. Separation preserves optionality across changing or heterogeneous objectives; embedding can only be preferred when decision-stage frictions (e.g., cognitive costs, defaults) make downstream implementation of optimal rules infeasible.

Key Points

• Conceptual framing

  • A trained predictor is represented by the distribution of posterior beliefs it induces (distribution-over-posteriors). Bayes-plausibility fixes the mean.
  • The relevant information order is the convex order (mean-preserving spreads).
  • Decision-stage value V(q) (optimal expected payoff given belief q) is convex in q, so more dispersed posteriors are always (weakly) better for any expected-utility decision problem.

• Preference embedding vs separation

  • “Preference embedding” = encoding asymmetric costs or objectives in the training loss (e.g., class-weighted log loss); “separation” = train with a strictly proper scoring rule to elicit calibrated probabilities, then apply preferences via an ex post decision rule (thresholding, cost-sensitive mapping).
  • Embedding preferences in the loss changes what the model learns (not just how reports are evaluated). Under realistic learning frictions (regularisation, capacity, optimisation), embedding often flattens incentives to learn distinctions near decision-relevant regions, pulling posteriors toward the mean.
  • The core primitive (diminishing-value-of-information condition): relative to a fixed preference-free loss, adding preference weights reduces the marginal value of extra informativeness; this generates a mean-preserving contraction of the posterior distribution (Theorem 1).

• Welfare consequence (robust separation principle)

  • Because V(q) is convex, a mean-preserving contraction reduces expected V. Thus, training with a preference-free strictly proper scoring rule and implementing preferences ex post weakly dominates preference-embedded training across all expected-utility decision problems (Theorem 2).
  • This result is decision-problem-agnostic: it holds without specifying downstream objectives.

• Exceptions and qualifications

  • Decision-stage frictions: if using the richer probabilistic signal is itself costly (e.g., cognitive costs, rational inattention, inability to compute optimal thresholds), embedding preferences upstream can be welfare-improving by compressing the signal and automating decision thresholds. The paper characterises when this “reversal” arises (Theorem 3).
  • Choice among preference-free losses matters: the class of strictly proper scoring rules is broad; under learning frictions, Bayes-risk curvature interacts with C(·) so one proper loss may be better than another in practice (Section 4; ordering of preference-free losses).
  • Scaling/normalisation matters for weighted losses: in class-weighted cross-entropy, Bayes-risk curvature can change with weight normalisation; for certain normalisations embedding can in principle increase informativeness rather than contract it (Proposition 1).

• RLHF and foundation models

  • Reinforcement learning from human feedback (RLHF) can be interpreted as exponential tilting of an underlying posterior-quality distribution.
  • If deployment objectives are stable and aligned, RLHF raises average quality and can improve welfare (Theorem 4).
  • When objectives are uncertain/multi-dimensional, concentrating model generation on one fixed weighting imposes a capability tax and reduces flexibility across objectives (Theorem 5), and it can amplify harms from reward misspecification (Proposition 6).

Data & Methods

• Nature of contribution: theoretical/analytical (no empirical dataset). Methods draw on:
  • Bayesian decision theory (indirect value V(q) and convexity).
  • Information design / Bayesian persuasion machinery (distribution-over-posteriors representation; Bayes plausibility).
  • Convex order comparisons (mean-preserving spreads/contractions); uses results and techniques from Strack and Yang (2024).
  • Proper scoring rules and Bayes risk theory (Gneiting & Raftery, 2007) to characterise preference-free losses.
  • Reduced-form cost-based learning model: for a given loss L, the learned posterior QL is any minimiser of E[HL(Q)] + C(Q) over Bayes-plausible Q, where HL is the Bayes risk of L and C is a law-invariant learning-friction functional capturing regularisation, limited capacity, optimisation frictions, etc. (Assumption 1, Eq. 12).
• Main formal results (among others):
  • Lemma: V(q) convex in q; hence more informative (in convex order) posteriors increase expected V.
  • Theorem 1: Under a diminishing-value-of-information condition, preference embedding induces a mean-preserving contraction of the posterior distribution.
  • Theorem 2 (Separation principle): Preference-free training with strictly proper scoring rules + ex post preference implementation weakly dominates embedded training across all expected-utility decision problems.
  • Theorem 3: Characterisation of when decision-stage (cognitive) frictions reverse the ordering.
  • Theorems 4–5, Proposition 6: Formal results on RLHF effects (tilting, capability tax, amplification of misspecification).
  • Existence and regularity: Proposition 2 gives mild conditions (continuity of H, law-invariance and lower semicontinuity of C, etc.) that guarantee a minimiser exists for the reduced-form learning problem.
• Key assumptions/limitations:
  • Core exposition focused on binary outcomes (scalar belief). Extensions to higher-dimensional/multi-class settings require additional structure (addressed in Section 6 / Appendix).
  • The learning friction C(·) is reduced-form and law-invariant; results rely on its properties (e.g., increasing cost for more informative Q).
  • The diminishing-value condition (Assumption 3) is pivotal; it must be checked for specific embedded losses and normalisations.

Implications for AI Economics

• Design guidance for AI pipelines

  • Prefer modular architectures: train models to produce calibrated probabilities using strictly proper scoring rules, and implement asymmetric costs or safety priorities via transparent downstream decision rules. This preserves option value and adaptability across objectives (regulatory changes, heterogeneous users, evolving safety norms).
  • Avoid baking user-specific asymmetric weights into core training objectives unless you are sure objectives are stable and decision-stage frictions dominate.
  • Ensure appropriate engineering to support post-processing: provide calibrated probability outputs, user interfaces or automation for applying optimal thresholds, and documentation about downstream decision rules.

• Practical diagnostics and checks

  • Measure posterior dispersion and calibration (beyond scalar accuracy): look for mean-preserving contraction effects when switching from preference-free to preference-weighted training.
  • Test sensitivity to loss normalisation and regularisation: embedding weights interact with penalty terms (weight decay, early stopping); do robustness checks across regularisation strengths.
  • Evaluate whether decision-stage frictions are binding: if users systematically use default cutoffs, struggle to compute thresholds, or exhibit cognitive limits, embedding or decision-support automation may be warranted.

• Policy and organisational implications

  • For regulators and institutions: require or encourage probabilistic outputs and transparency about post-processing rules where feasible, to maintain flexibility and enable audits across potential objectives.
  • In procurement and governance of foundation models: be cautious about permanently embedding single-objective RLHF at the expense of downstream flexibility; consider maintaining capability-preserving interfaces (probabilistic scoring or diversified evaluations) and modular reward tuning for specific deployments.

• Research directions in AI economics

  • Empirical work: quantify the capability tax in practical ML systems (how much posterior dispersion is lost under common weighted losses and regularisation regimes).
  • Multi-dimensional extension: explore convex-order analogues and sufficient structure for richer outcome spaces (multi-class, structured outputs).
  • Behavioral/organizational frictions: measure how real decision-makers use probabilistic outputs and when cognitive costs make embedding preferable; design interfaces that lower decision-stage frictions.

Caveats - The main results are theoretical and hinge on modelling choices (binary belief representation, form of learning friction C, and the diminishing-value condition). Applying the guidance requires empirical validation in specific ML pipelines and user contexts.