A Bayesian multi-LLM orchestration cuts simulated hiring pipeline costs by roughly one-third and slashes demographic parity gaps in a 1,000-resume test. Most of the savings come from aggregating diverse LLMs and updating beliefs sequentially, with targeted information gathering providing additional benefit.

Main Finding

Treating multiple LLMs as approximate likelihood providers and performing explicit Bayesian inference (eliciting p(evidence | state) via contrastive prompts, aggregating across models, applying explicit priors, and doing VOI-driven sequential updating) yields large, provable improvements for cost-sensitive, sequential decision-making. In a resume‑screening instantiation this approach cut total costs by 34% (saving $294k) vs the best single‑LLM baseline and substantially reduced demographic disparities (max group difference from 22 to 5 percentage points).

Key Points

• Diagnosis of failure mode: standard LLM agent pipelines treat models as discriminative classifiers that output p(state | evidence) and apply thresholds. The paper proves fundamental limitations of that paradigm for sequential decision problems:
  • Theorem (Sequential Updating Impossibility): you cannot compute a posterior after new evidence p(s | x1, x2) from a discriminative model's p(s | x1) and x2 alone (without re-querying on the joint input), because the likelihood p(x2 | s, x1) is missing.
  • Discriminative posteriors hide and entangle priors learned from training data, which can mismatch deployment base rates and systematically bias decisions.
  • Discriminative/batch querying prevents principled value‑of‑information (VOI) calculations necessary to decide whether to acquire costly additional information.
• Proposed alternative: treat each LLM as providing approximate likelihoods p(evidence | state) rather than posterior scores. Key steps:
  • Elicit p(x | s) via contrastive prompting for each hypothesized state s.
  • Aggregate across multiple diverse LLMs using robust statistics to form an ensemble likelihood.
  • Combine ensemble likelihoods with explicit, domain‑appropriate priors and apply Bayes’ rule for sequential belief updating.
  • Use expected‑utility maximization (incorporating asymmetrical costs) to choose actions and VOI calculations to decide whether to acquire additional information (phone screens, tests, authentication).
  • Use multi‑LLM ensembling to detect and mitigate model‑specific bias, improving fairness metrics.
• Empirical results (resume screening example):
  • Dataset: 1,000 resumes evaluated.
  • LLM ensemble: GPT‑4o, Claude 3.5 Sonnet, Gemini Pro, Grok, DeepSeek.
  • Action cost model: false hire / missed qualified = $40,000; onsite interview = $2,500; phone screen = $150.
  • Outcome: total cost reduction of $294,000 (34% improvement) relative to best single‑LLM baseline; demographic parity improved (max group difference down from 22 to 5 percentage points; 45% improvement).
  • Ablations: multi‑LLM aggregation accounted for 51% of cost savings, sequential updating 43%, and disagreement‑triggered information gathering 20% (ablation overlaps reflect combined effects).
• Additional claimed benefits: principled VOI enables cost‑effective information acquisition; explicit priors allow deployment‑appropriate calibration; ensemble likelihoods reduce disparate impact.

Data & Methods

• Application case: automated resume screening with a three‑stage decision space (resume → optional phone screen → onsite interview/hire/reject).
• Dataset: 1,000 anonymized resumes (paper provides domain assumptions about base rates and demographic groups for fairness evaluation).
• LLMs used: GPT‑4o, Claude 3.5 Sonnet, Gemini Pro, Grok, DeepSeek (selected for diversity).
• Likelihood elicitation:
  • Contrastive prompting: for each candidate state s (e.g., unqualified, borderline, qualified, exceptional), prompt models to estimate the probability of observing the resume or additional evidence given that state (p(x | s)).
  • Robust aggregation: combine per‑model likelihood estimates using robust statistics (e.g., median, trimmed mean, or heavy‑tailed aggregation) to reduce model‑specific bias and outliers.
• Bayesian pipeline:
  • Start with explicit deployment priors p(s) (calibrated to known base rates).
  • Compute posterior p(s | evidence) ∝ p(evidence | s) · p(s).
  • Sequential updates: when new evidence arrives (e.g., phone screen), use the elicited likelihood for that observation to update posteriors without re‑querying joint inputs.
• Decision rule and VOI:
  • Expected‑utility maximization with asymmetric cost function (domain cost matrix provided).
  • Compute VOI for potential information acquisition as expected reduction in decision cost minus information cost; acquire info if VOI > cost.
• Fairness mitigation:
  • Inspect ensemble likelihoods by demographic group; apply statistically principled recalibration or re‑weighting if systematic differences persist to reduce selection‑rate disparities.
• Evaluation metrics:
  • Total monetary cost (sum of action costs and error costs).
  • Fairness: maximum selection‑rate difference across demographic groups (demographic parity proxy).
  • Ablation: remove components (multi‑LLM aggregation, sequential updating, info‑triggering) to estimate contribution to savings.

Implications for AI Economics

• Economic value of principled inference: explicitly modeling likelihoods and priors and computing VOI can generate substantial monetary savings in high‑volume, high‑cost decision pipelines (the paper reports a 34% reduction in the hiring case). For firms processing large applicant or transaction volumes, such gains scale linearly with throughput.
• Resource allocation and optimal information purchase: VOI‑based acquisition provides a formal way to trade off inexpensive, incremental information (phone screens, tests, extra authentication) against expensive downstream mistakes, enabling firms to optimize operational budgets and staffing.
• Costly consequences of hidden priors: the paper highlights that model priors learned from training corpora can create systematic over‑ or under‑selection; correcting these via explicit priors avoids inflated downstream process costs (e.g., interview fatigue, lower hire quality).
• Fairness as an economic constraint: ensemble likelihoods plus explicit priors allow operators to reduce disparate impacts while still preserving informativeness, thereby lowering legal/regulatory risk and potential liability costs—important in regulated domains (employment, lending, healthcare).
• Role of multi‑LLM ensembles: combining diverse models acts as a form of model insurance—reducing variance, bias, and tail errors—and can be economically justified when the marginal cost of extra LLM queries is outweighed by avoided error costs.
• Deployment considerations and open practical trade‑offs:
  • Eliciting p(x | s) reliably from LLMs and aggregating across models imposes extra API and computation costs; firms must compare these costs vs expected error reductions.
  • Accurate domain priors and realistic cost models are essential; mis‑specified priors or cost matrices can negate benefits.
  • Legal/regulatory acceptance may favor transparent, decision‑theoretic pipelines that separate likelihoods and priors (better auditability), but claims about model‑elicited likelihood fidelity warrant further validation in each domain.
• Broader implication: AI economics should shift from accuracy‑centric evaluations to decision‑theoretic metrics (expected monetary/social cost, VOI, fairness constraints). Architectures that expose or approximate generative components enable economically and socially superior policies in sequential, cost‑sensitive settings.

If you want, I can extract the concrete decision‑rule formulas used in the paper (posterior update, VOI expression, expected utility calculation) and summarize the experimental protocol (prompts, aggregation rules, statistical tests) in more technical detail.