Counting compute flips leaderboard outcomes: when latency, tokens and API calls are penalized, models that buy small factuality gains via brute-force tactics (e.g., Best-of-N) lose to leaner agents—MAS-HQ exposes this deployment-relevant trade-off with a reproducible Q-Score.

Main Finding

Static factuality leaderboards that treat compute as free can mislead deployment choices: agents that “buy” marginal factuality with much higher compute (e.g., Best-of‑N sampling) can top static leaderboards yet be inferior once operational cost is counted. The paper introduces MAS‑HQ, a resource‑aware head‑to‑head evaluation protocol and a reference Q‑Agent that jointly scores factuality and normalized cost (the Q‑Score). Across summarization and open‑domain QA on news passages, MAS‑HQ reveals small but reproducible ranking reversals and pushes agents toward more resource‑efficient policies—even for frontier models (GPT‑5, Gemini‑2.5‑Pro).

Key Points

• Problem framed: leaderboards score factuality in isolation and implicitly assume infinite compute; this rewards brute‑force strategies (e.g., Best‑of‑N) that can be costly to deploy.
• MAS‑HQ protocol: pairs agents in competitive matches, measures factuality with any plugged‑in detector (H‑Score), computes a normalized operational cost penalty, and combines them into the Q‑Score: Q‑Score = (1/N) * sum_i (α * H‑Score_i − β * P_i) where P_i aggregates normalized API calls, tokens (input+output), review iterations, and wall‑clock time over the agents in the match. α and β are deployment knobs.
• Q‑Agent reference architecture: modular components (Policy Agent, Summarization Agent, Review Agent, Evaluation Agent) and an optional “vision mechanism” that reveals limited opponent telemetry when costly actions (e.g., review) are taken.
• Primary empirical finding: a Best‑of‑4 baseline had higher raw factuality (H‑Score 0.9169) than a Q‑Agent (H‑Score 0.9103) but lost on Q‑Score (0.5169 vs. 0.5217) while spending ~4× tokens and latency. The reversal is reproducible across 100 trials.
• The cost‑of‑knowing trade‑off persists for SOTA models (GPT‑5, Gemini‑2.5‑Pro): higher H‑Score does not guarantee a win once normalized cost is scored.
• Sensitivity and ablations: β tunes between raw, resource‑agnostic ranking (β → 0) and cost‑dominated behavior (large β). β = 0.01 was used as an intermediate illustrative setting; β = 0 collapses to static leaderboard behavior. MAS‑HQ is metric‑agnostic (quality detector can be replaced) and scales to >2 agents.
• Protocol design choice: cost penalty is relative (max‑normalized within a match), so Q‑Score is match‑specific; intrinsic rankings would require tournament/Elo aggregation (left for future work).

Data & Methods

• Tasks and data: evaluation on >1,000 long‑form news passages used as information‑synthesis proxies (808–813 passages evaluated per head‑to‑head match after filtering for model safety refusals).
• Factuality metric: automated discriminator from the hallucination leaderboard (Bao et al., 2024). Human validation: blind expert evaluation on 100 randomly sampled summaries produced Pearson r = 0.96 with the discriminator H‑Score.
• Q‑Score components:
  • H‑Score_i ∈ [0,1] for each task i (from the plugged detector).
  • P_i = normalized penalty aggregating API calls, total tokens (input+output), review iterations, and runtime; max‑normalized within the match.
  • α, β > 0 are deployment parameters (α = 1, β = 0.01 used as the main reported point; sweeps performed).
• Q‑Agent implementation:
  • Policy Agent (PA): chooses actions {continue, review, end} given private state and optional opponent telemetry.
  • Summarization Agent (SA): generates initial summary (not directly cost‑aware to avoid trivial "say less" shortcuts).
  • Review Agent (RA): optional refinements, consumes review budget R (R = 3 in experiments).
  • Evaluation Agent (EA): computes H‑Score and tracks resource use to inform PA decisions.
  • Vision mechanism: parameterized telemetry knob; when enabled, costly actions reveal a snapshot of opponent state to the rival.
• Baselines and comparisons: single‑agent Best‑of‑N (N=1,2,4) baselines; multiple LLM families tested (GPT‑4o‑mini, GPT‑5, Gemini‑2.x, Grok‑3, Qwen‑Max, Deepseek‑v3).
• Reproducibility checks: repeated matches (100 trials) to assess variance; ablations for β, review component, vision knob; also tested replacing Q‑Agent with standard ReAct agent under same rules.

Implications for AI Economics

• Model selection and procurement: MAS‑HQ demonstrates that pure accuracy rankings can be misleading for deployment decisions. Procurement should consider the marginal cost of factuality (compute, latency, API calls), not only raw accuracy.
• Pricing and SLAs: α and β map to deployment SLAs and cost–quality trade‑offs. Buyers and providers can use MAS‑HQ‑style curves to set SLA parameters, price API tiers, or choose models that optimize operational utility given latency/token budgets.
• Incentives for system design: scoring cost alongside accuracy discourages brute‑force, resource‑heavy mitigation tactics (e.g., large Best‑of‑N runs) and incentivizes more efficient verification/revision policies or architectural changes (e.g., cheaper retrieval pipelines, selective review).
• Market signals and competition: head‑to‑head, match‑relative penalties simulate competitive deployment settings where cost is relative; this can change ordering among providers and inform competitive strategy (e.g., optimize for cost‑adjusted utility rather than raw H‑Score).
• Capacity planning and cost‑benefit analysis: MAS‑HQ produces reproducible, quantifiable trade‑off estimates that can feed capacity planning, budget allocation, and marginal cost calculations for real‑time systems (finance, news intelligence) where latency matters.
• Limitations and further economic modeling:
  • Q‑Score is match‑specific because penalties are normalized across co‑present competitors; deriving an intrinsic, market‑wide ranking requires tournaments or Elo‑style aggregation.
  • The protocol measures operational proxies (tokens, API calls, runtime), not direct monetary cost; mapping to $ cost requires provider pricing data.
  • Results depend on the chosen factuality detector and α/β settings; detector reliability and SLA calibration are economic inputs to the evaluation.
  • Additional externalities (e.g., user trust, monetized errors, downstream costs of hallucination) are not directly priced in the Q‑Score and would need domain‑specific extensions.
• Practical takeaways for economists and practitioners:
  • Use resource‑aware evaluation when making deployment or procurement choices.
  • Report cost‑sensitivity (α, β) and provide cost‑accuracy curves rather than single static metrics.
  • Consider tournament aggregation to produce market‑level rankings and incorporate provider pricing to translate normalized penalties into monetary terms.

Summary: MAS‑HQ operationalizes “cost of knowing” by combining any factuality detector with a normalized operational cost penalty in a competitive match format. The protocol surfaces deployment‑relevant trade‑offs, punishes brute‑force, expensive strategies, and yields reproducible ranking changes that static leaderboards hide—insights directly relevant to AI procurement, SLA design, and economic evaluation of LLM systems.