AI models optimized for raw accuracy can be the wrong product: firms should choose how often models predict (coverage) to fit users’ verification costs and error losses, and small changes in those downstream economics can cause abrupt shifts in the optimal training target.

Main Finding

The economic value of an AI prediction depends on how humans use that prediction inside a larger decision process (the “composite experiment”). Maximizing unconditional accuracy (overall percent correct) is generally not the right training objective. Instead, optimal model training trades off coverage (probability the model issues a prediction) against conditional correctness (probability the prediction is correct given the model predicts), and the downstream human continuation plan picks the tangency point on a model’s accuracy–coverage frontier. Because users may switch between qualitatively different continuation plans (e.g., verify vs trust), small changes in economic parameters can produce discontinuous jumps in the optimal training target.

Key Points

• Basic primitives and notation
  • Model m = (q, p, c): coverage q = Pr(model predicts), conditional correctness p = Pr(prediction correct | model predicts), query cost c. Unconditional accuracy a = q p.
  • Agent payoffs: correct label gives H > 0, incorrect gives −L < 0, outside option normalized to 0.
• Composite experiment and continuation plans
  • A continuation plan is the agent’s full contingent decision rule after observing the model (and possibly additional actions such as verification or consulting a second model).
  • Under a coverage-invariance assumption (the downstream continuation signals and costs conditional on coarse AI outcomes do not depend on (q,a)), the payoff from any continuation plan is affine in (q,a).
  • The agent’s value is the upper envelope (convex, piecewise-linear) of these affine plan payoffs and the baseline.
• Verification example (pedagogical core)
  • With a perfect verification cost cV, the undominated downstream plans are: don't query; query and trust the model; query and verify (act only if verified).
  • Plan payoffs reduce to affine forms like Π(q,a) = α + β a − γ q − c, with slopes determined by H, L, cV.
• Training frontier and tangency principle
  • Training yields a frontier of feasible (q,p) (or equivalently of attainable unconditional accuracy a as a function of q). Typically assumed concave: attaining higher coverage requires accepting lower conditional correctness.
  • For a given continuation plan (an affine payoff), optimal training is found at the tangency between that plan’s iso-payoff line and the accuracy–coverage frontier.
  • When the agent’s optimal continuation plan changes (e.g., verification becomes too costly and the user switches from verify→trust), the tangency can jump to a different point on the frontier → discontinuous changes in the optimal (q,p) target.
• Consequences and comparative statics
  • Environments with cheap verification (or low downside L) favor high coverage: catch more cases and verify cheaply.
  • Environments with expensive verification and high L favor selective models with low coverage but high conditional correctness.
  • Heterogeneous users served by a single trained model create welfare losses; monopolists optimally target marginal users rather than population-average welfare and may strategically offer coarser outputs (e.g., subscription tiers).
• Connections to ML literature
  • Relates to selective classification / reject option and learning-to-defer literatures; difference: this paper takes the continuation technology and agent payoffs as primitives and asks where on the training frontier (q,p) the model should operate.
  • Also relates to Blackwell comparisons of experiments, but Blackwell orderings need not hold here because of coarse outputs and the coverage–accuracy tradeoff.
• Robustness / scope
  • Main results hold for a broad class of finite, coverage-invariant continuation technologies (multiple verification strategies, partially correlated alternative models, etc.). Infinite policy classes need extra regularity.
  • Baseline assumes model outputs are coarse (predict label or abstain); allowing calibrated case-specific posteriors changes the trainer’s problem (possible to reveal quality instead of choosing coverage).

Data & Methods

• Methodology: analytic, theoretical model and comparative-statics. No empirical estimation or dataset used.
• Core formal steps:
  • Define primitives (q, p, a, H, L, costs) and the agent’s sequential decision problem (composite experiment).
  • Show continuation payoffs Πr(q,a) are affine under coverage-invariance (Lemma/Proposition).
  • Define a model training frontier mapping feasible conditional correctness to coverage (concave frontier commonly assumed).
  • Use geometric/tangency arguments: for each affine continuation payoff, the optimal (q,a) solves a tangency between an iso-payoff line and the frontier; agent’s value is the upper envelope over plans.
  • Analyze comparative statics and show possible discontinuities when the agent switches continuation plans.
  • Extend framework to consider heterogeneous users, monopolist model trainer incentives, subscription/coarse-output strategies, team verification, adversarial settings.
• Key assumptions to note:
  • Coverage-invariance: downstream signals and costs, conditional on coarse events (no-prediction, correct, wrong), do not change with the model’s global (q,a).
  • Coarse outputs: model gives a label or abstains rather than calibrated posteriors (baseline).
  • Risk-neutral agents; finite set of continuation plans (for formal piecewise-linear envelope argument).

Implications for AI Economics

• For model designers and firms
  • Training objectives must incorporate downstream human behavior and verification costs. Maximizing unconditional accuracy is not generally welfare-maximizing.
  • Decide whether to prioritize coverage (more predictions, lower per-prediction accuracy) or selectivity (abstain on hard cases): criterion depends on H, L, verification costs, and the human continuation technology.
  • Firms (especially monopolists) may strategically choose model coarseness or coverage to target marginal consumers and may offer tiered/coarse outputs for price discrimination.
• For practitioners and procurement/evaluation
  • Evaluate models by their value inside realistic composite experiments (how will users verify, defer, or trust?), not only by global accuracy metrics.
  • When deploying predictive systems in domains with cheap verification (e.g., cheap tests, code testing), favor broader-coverage models; where verification is costly and errors are costly (e.g., high-stakes medical/safety decisions), favor selective models or provide calibrated uncertainty.
  • Be alert to discontinuities: small changes in verification costs or loss parameters (or policy/regulation) can justify materially different training targets and deployment rules.
• For policy and regulation
  • Standards and procurement specifications should reflect the downstream decision environment (verification capacity, error costs) rather than raw accuracy comparisons.
  • Encouraging or enabling verification capacity (e.g., cheaper confirmatory tests, better human-AI workflows) can change the societal optimal design of AI systems toward higher coverage and broader usefulness.
• For research
  • Empirical work: measure verification costs, downstream loss magnitudes, and how users switch between continuation plans in real settings—these parameters determine the optimal modeling tradeoffs.
  • Theoretical extensions: relax coverage-invariance, allow calibrated/posterior outputs, endogenize verification technology investments, and consider infinite (or continuous) policy classes for richer applications.
  • ML methodology: integrate downstream payoff structure into training objectives or selection among selective classifiers / abstention rules; study methods to reveal case-level uncertainty (calibration) so that coverage need not be hard-coded by the trainer.

Short takeaway: pick the training target (coverage vs accuracy) with the human decision environment in mind. The best AI for welfare is the one that, together with user verification and follow-up options, produces the best composite experiment — not necessarily the one with the highest raw accuracy.