Conventional winrate metrics nudge model makers toward sameness and hurt users; a simple 'weighted winrate' that credits higher-quality answers realigns incentives, promoting specialization and raising consumer welfare in theory and benchmark experiments.

Main Finding

When users noisily aggregate across models (modeled by a Bradley–Terry–Luce/Boltzmann choice model), standard producer incentives based on winrate (probability a model is selected) push producers to homogenize their models, which reduces consumer welfare. A simple alternative objective—weighted winrate (the product of selection probability and the value of the chosen response)—better aligns producer incentives with consumer welfare: it provably encourages specialization (complementary strengths across tasks) and increases consumer utility. The paper also shows surprising non‑monotonicities: adding or improving a model can sometimes reduce aggregate utility under noisy aggregation.

Key Points

• Model of the marketplace
  • There are T discrete tasks and M models. Each model m has deterministic (or expected) values vm,t for task t.
  • Consumers choose a model per task according to a choice function; expected task utility is the sum over models of vm,t · p_m(⃗v_t).
• Choice/aggregation model
  • Focus on Bradley–Terry–Luce (BTL) / softmax choice with temperature β. BTL interpolates between random choice (high β) and perfect selection (low β).
  • BTL captures the intuition that users/routers are more likely to confuse models of similar value.
• Consumer welfare vs producer objective
  • Consumer welfare: sum over tasks of expected value under the choice rule.
  • Typical producer objective (status quo): maximize winrate (probability of being selected / market share).
• Non‑monotonicity results
  • Under BTL aggregation, adding a new model or replacing an existing model can have counterintuitive effects:
    • A new model that is better on average than all existing models can nevertheless reduce consumer welfare once added.
    • A strictly better model replacement (taskwise improvements) can sometimes reduce aggregate utility (monotonicity can fail).
  • These effects depend on the BTL temperature β and the complementarity structure across tasks.
• Incentives and homogenization
  • If producers compete to maximize winrate, equilibria tend to involve models becoming similar across tasks (homogenization). This is because winrate rewards being good where others are good (to capture selection probability) rather than specializing where you uniquely help aggregation.
  • Homogenization reduces the gains from aggregation/routers and thus lowers consumer welfare relative to a specialized ecosystem.
• Weighted winrate mechanism
  • Definition: for a model on a task, score = p_m(⃗v_t) · v_m,t (probability picked times its value).
  • Intuition: rewards both being chosen and providing higher value, akin to revenue = price × market share.
  • Theoretical results: weighted winrate leads producers to prefer specializing on particular tasks and provably improves incentives toward higher consumer welfare relative to plain winrate.
• Empirical exploration
  • The paper includes experiments on benchmark datasets / simulated markets (Section 7) to explore how theoretical results manifest in practice; results support the qualitative claims (effect sizes and calibration depend on datasets and routing noise).

Data & Methods

• Formal/theoretical framework
  • Tasks: T ≥ 2 discrete tasks. Models characterized by vectors {v_m,t}. Total/average values used in proofs.
  • Aggregation: consumer selects model per task via a choice function; main focus on BTL (softmax) with temperature β.
  • Producer actions: model creation (introduce a new model with chosen profile given a cap) and model replacement (modify an existing model’s task performance).
  • Objectives: consumer welfare (expected task values under aggregation); producer objective either winrate (p_m) or weighted winrate (p_m · v_m,t).
  • Analysis: constructive examples, lemmas and theorems with proofs (appendix), comparative statics in β, equilibrium reasoning about producer best responses.
• Empirical/simulation methods (Section 7)
  • Simulated/real benchmark model pools and routing/noise experiments to demonstrate the non‑monotonicities and incentive shifts in realistic settings.
  • Calibration of β and modeling of user selection/routing noise to assess robustness.
• Assumptions & scope
  • Values vm,t are deterministic or treated as expected values.
  • Consumers are rational according to BTL (or explored alternatives); producers optimize average winrate (status quo) or weighted winrate.
  • Leaderboard/market interventions are feasible (especially in ML evaluation settings where metrics can be changed).

Implications for AI Economics

• For leaderboard and marketplace design
  • Current popularity of winrate-based metrics (or ranking by selection frequency) can incentivize producers to converge to similar, “generalist” models that capture selection probability rather than complementary strengths — reducing the ecosystem value for consumers who could benefit from aggregation.
  • Leaderboards and routing evaluations should consider weighted metrics (e.g., weighted winrate) that combine selection probability with measured quality to induce diversity/specialization.
• For routing & aggregator design
  • Router/noise level matters: when routing is imperfect (higher β), non‑monotonic harms are more likely. Improving routing accuracy can mitigate some harms but does not replace the need for proper producer incentives.
  • Aggregators (automated routers or human selection interfaces) should be designed to elicit not just a preferred model but also a quality signal (ratings, scores) to enable weighted incentives.
• For producers and competition policy
  • Incentive design matters: simple market‑share incentives favor copying the market center; payoffs that scale with delivered value encourage producers to target niche strengths that improve overall welfare when aggregated.
  • Mechanism interventions (incentive schemes, curated evaluation metrics) can be effective levers to shape model diversity without altering underlying technology.
• Practical considerations & limitations
  • Eliciting truthful value assessments from users (to compute weighted winrate) is nontrivial; coarse ratings or proxy metrics could be used but open questions remain about bias, calibration, and strategic reporting.
  • Theoretical results depend on the BTL choice model and other modeling choices (deterministic vm,t, producers’ objectives); real markets may exhibit additional frictions (costs to specialization, multi‑period dynamics, strategic submissions).
  • Weighted winrate could be gamed if not carefully implemented; leaderboard design must consider manipulation resistance and measurement noise.
• Research and policy directions
  • Empirically validate the magnitude of these effects in deployed markets and measure how routing accuracy (β) interacts with incentives.
  • Design practical elicitation protocols for user quality signals suitable for weighted scoring (e.g., calibration methods, robustness to strategic reporting).
  • Study dynamic markets with entry/exit costs, multi‑period learning, and richer producer objectives to refine mechanism prescriptions.
  • Explore hybrid metrics and policy tools (subsidies, curated benchmark sets) to promote a healthy, diverse model ecosystem that maximizes consumer welfare.

Summary takeaways: noisy aggregation can create perverse incentives under common winrate metrics, producing homogenized model markets that lower aggregate user value. A simple, interpretable fix—weighted winrate—better aligns producers with consumer welfare by rewarding both selection and the intrinsic quality of chosen outputs; practical deployment will require careful measurement and anti‑gaming design.