Correlated errors across language models limit the benefits of majority voting and alter the optimal approval threshold; modeling state-dependent dependence on held-out items cuts screening loss by about 15.7% relative to majority voting and gives modest additional gains over independence-based thresholding.

Main Finding

State-dependent correlations in AI agents’ binary errors materially change how committees should be designed. Shared (latent) error components produce a nonzero asymptotic error floor for majority voting, can collapse diverse-sounding committees toward single-agent behavior as correlation → 1, and—when estimated and modeled—lead to better held-out loss prediction and different, often stricter, cost‑sensitive voting thresholds than those chosen under an independence assumption.

Key Points

• Model and mechanism

  • Agents’ binary votes arise from latent scores Z_i = √ρ_θ W + √(1−ρ_θ) ε_i, where W is a common factor and ρ_θ is the state‑conditional latent correlation (θ ∈ {good, bad}). Votes are thresholded indicators of Z_i.
  • Dependence is state‑conditional: correlations can differ on good versus bad items.
  • Loss is normalized expected cost: false negatives plus κ × false positives, where κ encodes prior × cost tradeoffs (Sah–Stiglitz screening setup).

• Analytic results

  • Correlation weakens canonical protections of extreme rules: hierarchies (unanimity) become worse at filtering bad proposals as ρ_B increases; polyarchies (any approval) become worse at rescuing good proposals as ρ_G increases.
  • Majority voting no longer drives error to zero with large committees. For majority, the infinite-committee residual error is ε∞(p,ρ) = Φ(−Φ⁻¹(p)/√ρ) > 0, so positive ρ leaves a positive fraction of single-agent error irreducible.
  • As ρ → 1 (for fixed n), any k-of-n rule converges to single‑agent behavior (committee collapse).
  • The cost-weighted optimal threshold depends jointly on marginals (p_G, p_B), prevalence, κ, and the state‑conditional correlation structure; unequal ρ_G vs ρ_B can change the monotone likelihood ordering and thus the optimal k.

• Empirical validation (held‑out)

  • Dataset: 174,384 binary votes from 28 language‑model judge units (20 open‑weight models × 3 sampling runs + API models) across four labeled screening tasks: fact verification (400 items), math solution verification (400, half corrupted), code review (357, 157 mutated), truthfulness screening (400).
  • Estimation: per-agent p_G, p_B and pairwise tetrachoric latent correlations (ρ_G, ρ_B) estimated on odd-indexed items; evaluated on even-indexed items. Correlation matrices were projected to be positive semidefinite when needed.
  • Prediction performance: heterogeneous full‑matrix Gaussian‑copula model predicted held‑out committee losses with identity-line R² = 0.967 (RMSE 0.088). Independence baseline: R² = 0.840; exchangeable one‑factor projection: R² = 0.906.
  • Design‑balanced selection inference (20 balanced cells, bootstrap): multiplied-loss means (×100) — majority: 60.25; independence‑based optimal threshold: 52.50 (improvement ≈ 7.75 units, 95% CI 6.34–8.86); dependence‑aware threshold: 50.77 (further improvement ≈ 1.73 units, 95% CI 0.68–2.33). Total reduction from majority ≈ 15.73% (95% CI 13.41–16.75%). Dependence modeling changed the selected threshold in ≈ 40.9% of weighted committee–κ cases.
  • Measured correlations: same‑model resamples ≈ ρ ≈ 1; open‑weight cross‑family pairs showed substantial positive ρ (median ≈ 0.58–0.60), with state dependence present.

• Practical caveats

  • Tetrachoric correlations are unidentifiable for nearly‑constant voters; estimates required boundary handling and PSD projection.
  • Empirical results are conditional on the four benchmarks and the specific roster of models used.

Data & Methods

• Data

  • Four binary screening domains with gold labels: fact verification (400), GSM8K math (400), MBPP code (357), TruthfulQA (400).
  • Judges: 28 model–sampling-run agents (20 open‑weight instruction‑tuned models with sampling variations + 8 API models); each judge provided 1 vote at temperature 0 and 3 votes at temperature 0.7 (treated as separate agents where appropriate), yielding 174,384 votes in total.

• Estimation

  • Split: odd‑indexed items for estimation, even‑indexed for evaluation.
  • For each agent, estimate state‑conditional approval probabilities p_G and p_B.
  • For each pair, estimate tetrachoric latent correlations ρ_G and ρ_B (subject to Fréchet bounds); assemble state‑specific correlation matrices and project to positive semidefinite cone when necessary.

• Modeling & validation

  • Heterogeneous Gaussian‑copula simulation: simulate correlated latent normals using the estimated marginals and full correlation matrices to compute acceptance probabilities A_{n,k}(p,ρ) and expected loss across k-of-n rules for varying κ ∈ {0.25, 0.5, 1, 2, 4}.
  • Baselines: independence (identity correlation matrix) and a nonnegative exchangeable one‑factor projection.
  • Validation: predict loss for every k-of-n rule on held‑out votes; compute identity-line R² and RMSE; perform bootstrap (1000 replicates with design balancing) for threshold selection inference.

Implications for AI Economics

• Procurement and roster design

  • Value of diversity: procurement should prioritize agents with complementary, not just individually strong, error patterns. Pricing and contracting should account for the marginal value of reduced state‑conditional dependence.
  • Diminishing returns to scale: because positive correlation imposes an asymptotic error floor, simply buying more copies or similar models yields limited gains—markets for ensemble services should price diversity and de‑correlation capabilities higher.

• Organizational decision rules and cost tradeoffs

  • Cost-sensitive aggregation: optimal approval thresholds depend on error costs (κ) and on state‑conditional correlations; organizations should estimate these inputs and choose k accordingly rather than defaulting to majority.
  • Risk management: correlated failures increase systemic risk of simultaneous misjudgment; regulators and firms should model dependence when relying on AI panels for high‑stakes screening.

• Product strategy and incentives

  • Incentivize orthogonality: model developers and platform buyers can improve system-level performance by investing in techniques that reduce shared blind spots (data diversification, architecture diversity, specialized fine‑tuning).
  • Differentiated markets: vendors that produce models with less correlated errors (particularly in the costly‑mistake state) may command premium value in ensemble applications.

• Evaluation and governance

  • Performance metrics must go beyond marginal accuracy: evaluate pairwise and state‑conditional correlations to predict ensemble behavior and set appropriate deployment thresholds.
  • Monitoring requirements: continual estimation of ρ_G, ρ_B and re‑optimization of thresholds as the roster or task distribution shifts.

• Research and policy directions

  • Further work should estimate dependence on broader task sets, larger and more heterogeneous rosters, and study dynamic/adaptive aggregation rules (beyond fixed k-of-n) that can exploit richer signals than vote counts.
  • Policy implications include standards for ensemble testing and transparency of model complementarities for public‑interest deployments.

Practical recommendation (operational): measure each candidate model’s p_G, p_B and pairwise tetrachoric correlations on a validation corpus, simulate the Gaussian‑copula ensemble to compute expected loss across k and κ values, and pick the k that minimizes cost‑weighted loss; prioritize acquiring models that reduce state‑conditional correlations most relevant to the costly error type.