Providers typically route tasks with a simple static rule, but misaligned cost and user-utility rankings can lead them to throttle or misroute models to cut costs, depressing user utility.

Main Finding

Routing decisions between two LLMs (a cheaper “standard” model M1 and a stronger “reasoning” model M2) can be solved as a Stackelberg game between a provider (who routes and chooses cascade rules) and a user (who may re-prompt or abandon). Closed-form analysis shows that in most parameter regimes the provider-optimal policy is a simple static routing decision (route to one model without cascading) determined by simple thresholds based on (a) users’ net-per-pass values and (b) providers’ cost-per-success trade-offs. Cascading (multi-pass escalation) is optimal only in limited parameter regions where models are sufficiently differentiated and abandonment penalties are in a particular range. Crucially, there exist persistent misalignment regimes where provider cost minimization routes users to a model that yields lower user utility, and providers may be incentivized to throttle latency to discourage retries when churn penalties are low.

Key Points

• Model summary
  • Two models: M1 (faster/cheaper) and M2 (slower/expensive), with success probabilities p1<p2, latencies t1<t2, costs c1<c2.
  • User gets value V on success; each pass imposes latency cost to the user; provider incurs inference cost per pass and faces penalty P when the user abandons (e.g., lost future revenue).
  • Define user net-per-pass ξi = V·pi − ti. Define provider cost-of-pass metric ~ ci/pi.
• User best response (given provider route i and cascade probability s)
  • If routed directly to M2: user stays iff ξ2 ≥ 0 (else abandons).
  • If routed to M1: behavior depends on signs of ξ1, ξ2:
    • If both ξ1, ξ2 ≥ 0: user always stays (q*=0).
    • If both ξ1, ξ2 ≤ 0: user always abandons (q*=1).
    • If ξ1 < 0 < ξ2: user stays only if cascade probability s exceeds a threshold s0; otherwise abandons.
    • If ξ1 > 0 > ξ2: there are three regions in s producing (stay, mixed stochastic abandonment, or abandon); interior region yields an interior abandonment probability q*(s).
• Provider optimal policy (assuming rational user)
  • Problem reduces to comparing a small set of scalar options: route to M1 (no cascade), route to M2 (no cascade), or a limited set of cascade choices.
  • When both models have same sign of ξ: always route to single best model (no cascade). If both value-dominated, route to model with lower cost-per-success (ci/pi). If both latency-dominated, route depends on penalty P vs incremental cost-of-pass.
  • When models differ in sign (one value-dominated, the other latency-dominated), the provider often still prefers routing to M1 (the cheaper model) except when cost-of-pass ordering or abandonment penalty crosses thresholds; cascading is optimal only in select parameter windows.
• Welfare / alignment insights
  • Misalignment: Providers minimize expected servicing cost + abandonment penalty, which can lead them to route to cheaper models even when users prefer the stronger model — especially when c1/p1 < c2/p2 and abandonment penalty is small.
  • Throttling incentive: If abandonment penalty P is low, providers can reduce expected costs by increasing latency (discouraging retries), which lowers user utility — an extreme misalignment possibility.
• Practical upshot: optimal rules are simple threshold rules in terms of ξi and cost-per-success ci/pi, making them actionable for providers but highlighting potential welfare losses absent corrective incentives.

Data & Methods

• Approach: Analytic, game-theoretic modeling and closed-form solution of a Stackelberg/Markov game between one provider and one (representative) user.
• Core model ingredients and assumptions:
  • Two LLMs (M1, M2) with scalar parameters (pi, ti, ci); t1<t2, c1<c2, p1<p2. Success per pass is Bernoulli and i.i.d. across passes.
  • Provider commits to an initial route i∈{1,2} and a cascade probability s∈[0,1] to escalate from M1→M2 upon failure.
  • User chooses an abandonment probability q upon each failure (stationary policy), maximizing expected V·(eventual success probability) minus cumulative latency.
  • Provider minimizes expected inference cost plus abandonment penalty P·Pr(abandon).
  • Closed-form expressions derived for user success probability, expected cumulative latency, provider expected cost; thresholds (s0, sL, sH, P1, P2) solved analytically.
• Analysis: Characterization of the user best response (q(i,s)) and substitution into provider objective to find provider-optimal (i, s*). Results illustrated with heatmaps/figures mapping regimes in ξ1–ξ2 and cost-of-pass and P dimensions.
• Limitations to note:
  • Single-provider, single-user setup (no competition, aggregate dynamics, or long-run subscription dynamics beyond per-task abandonment penalty).
  • IID success per pass and full user knowledge of routing/cascade policy (simplifies belief/update dynamics).
  • No learning/adaptation of p1,p2 or user heterogeneity modeled (extensions could relax these).

Implications for AI Economics

• Operational decisions can be reduced to a few interpretable metrics: user net-per-pass ξi = V·pi − ti and provider cost-per-success proxy ci/pi. These provide simple decision rules for routing.
• Welfare vs provider-cost trade-offs:
  • Provider cost minimization can systematically conflict with user welfare. Firms choosing cheaper models by ci/pi may degrade user utility when users value higher success probability (higher ξ).
  • Regulators or platform designers should be aware that “optimal” cost-saving routing can produce hidden user welfare losses and, if unchecked, longer-term churn not captured by a per-task P.
• Pricing and incentive alignment:
  • To align provider routing with user welfare, platforms can internalize longer-run churn risk (increase effective P in optimization), adjust subscription pricing, or offer explicit quality-tier pricing so user preferences map to revenue.
  • Alternatively, exposing routing signals or allowing user choice of model tier (with transparent cost trade-offs) could reduce misalignment.
• Cascading design guidance:
  • Cascades are only beneficial in bounded parameter regimes — specifically when models are sufficiently differentiated and abandonment penalties/ci/pi relationships lie in certain ranges. Generic use of cascades can add cost without improving welfare.
  • Providers should instrument and estimate ξi and ci/pi per workload to decide whether to cascade or route directly.
• Strategic throttling risk:
  • Providers might be incentivized to increase latency (or otherwise discourage retries) when abandonment penalties are low, reducing their costs at the expense of user utility. Monitoring and potential regulation of such incentives (or reputational constraints) may be warranted.
• Directions for empirical work and policy:
  • Empirically estimate per-query p, t, c and measure actual user V (task-dependent) and long-run churn sensitivity to validate regime boundaries.
  • Extend to multi-user, competitive-provider markets to study how competition mitigates or exacerbates misalignment and throttling incentives.
  • Consider contract designs (performance SLAs, quality-based pricing) to internalize user welfare in provider routing decisions.

Summary takeaway: Routing decisions for multi-model LLM services can be optimized with simple thresholds, but doing so solely on provider cost metrics risks persistent misalignment with user welfare; cascades are useful only in specific regimes, and low churn penalties create incentives to throttle latency, which harms users.