A multi-agent LLM pipeline framed as a finite-horizon MDP improves safety classification: up to 19% higher accuracy and as much as an 85x reduction in required human review versus a single-agent baseline in a self-harm detection compliance case study.

Main Finding

A structured multi-agent framework that models decomposable compliance workflows as a bounded‑horizon MDP over a directed acyclic process map — combined with per‑agent Monte Carlo sampling to estimate epistemic uncertainty — can improve labeling accuracy and greatly reduce required human review relative to a single‑agent baseline. In the authors' self‑harm detection case study (N=112), accuracy improved up to 19% and human review requirements fell by up to 85× in some configurations, while maintaining or reducing false positives.

Key Points

• Architecture
  • Workflow is a DAG G = (S, E) where nodes S are role‑specific LLM agents (Worker, Triage, Risk, Legal) and terminal states are {safe, unsafe, human review}.
  • The process is formalized as a bounded‑horizon MDP constrained by the DAG (all episodes terminate in ≤ τmax steps).
• Uncertainty quantification
  • Epistemic uncertainty is inferred per agent by Monte Carlo sampling: each agent emits n independent label samples a(1..n) ∈ {safe, unsafe, uncertain}.
  • System‑level uncertainty is captured by propagation through the MDP and termination in human‑review states.
  • Policies π map sample vectors to decisions (experiments used a static majority‑vote policy; thresholds can be tuned).
• Empirical results (AEGIS 2.0 self‑harm benchmark; N=112: 68 safe / 44 unsafe)
  • Baseline (single‑agent with CoT): accuracy ≈ 69.8% (±1.8).
  • Multi‑agent configurations (n samples): n=1 → 88.0% (±2.3); n=3 → 84.8% (±0.8); n=5 → 84.3% (±1.5).
  • Human review: baseline ≈ 17.2 cases; multi‑agent n=1 ≈ 0.2; n=3 ≈ 2.6; n=5 ≈ 4.2 (means with 95% CI reported).
  • Timing: baseline ≈ 17.7 s; multi‑agent n=1 ≈ 12.3 s; n=3 ≈ 42.1 s; n=5 ≈ 83.6 s — showing tradeoffs between compute, latency, and review reduction.
  • False positives stayed within baseline CI; sensitivity with larger n (e.g., n=25 on a false‑negative subset) eliminated false positives in that subset.
  • The multi‑agent logs helped identify annotation errors in the benchmark.
• Practical properties
  • Modular and interpretable: decision traces and agent‑level uncertainty point to which agent/escalation edge to tune.
  • Adaptable: human‑reviewed cases can be used to update transition probabilities P, agent prompts, and escalation policies π.
  • Supports batching and parallel execution for efficiency; extensible to domain‑specific foundation models.

Data & Methods

• Formal model
  • States: agents and terminal labels; actions: vectors of Monte Carlo label samples.
  • Transition probabilities P(st+1 | st, at) are restricted to DAG edges and estimated empirically from observed outputs.
  • Reward R includes correctness, escalation cost, and latency considerations.
  • Expected return Vπ(s, x) computed under policy π mapping sample vectors to final decisions.
• Experimental setup
  • Agents instantiated with SOP‑style prompts (Worker, Triage, Risk, Legal) using gpt‑5.
  • Policy π: static majority vote at each node during experiments (but authors note π can be optimized).
  • Monte Carlo sample sizes tested: n ∈ {1, 3, 5}; additional sensitivity with n = 25 on a subset.
  • Dataset: AEGIS 2.0 AI Safety Benchmark (N=112); experiments run 5× to produce confidence intervals.
• Metrics reported
  • Accuracy, number of human reviews, counts of false positives/negatives, and processing time per example.
• Limitations noted by authors
  • Small public benchmark (112 examples) and a fixed policy in experiments.
  • Use of a particular LLM (gpt‑5) and SOP prompts — results may vary with other models or prompt designs.
  • Need for empirical data to estimate transition probabilities; potential for model drift over time.

Implications for AI Economics

• Labor substitution and cost structure
  • Large reductions in human review (reported up to 85×) imply significant labor cost savings in compliance and content‑moderation settings, especially where human reviewers are expensive and scarce.
  • Tradeoff: savings in reviewer labor versus increased computation cost (larger n increases latency and compute). Optimal deployment requires comparing marginal compute costs to marginal reductions in human review and expected cost of errors.
• Value reallocation and task composition
  • Demand for routine review labor may shrink, while demand may increase for higher‑skill roles: process designers, SOP authors, model auditors, and oversight specialists who tune transition probabilities, policies, and interpret logs.
  • Firms can capture value by investing in process mapping, logging infrastructure, and dynamic retraining/monitoring to maintain performance and regulatory compliance.
• Risk, liability, and regulatory economics
  • Improved interpretability and auditable decision traces lower regulatory friction and liability uncertainty, potentially reducing compliance costs and easing adoption in regulated sectors (healthcare, finance).
  • Quantifying and controlling uncertainty can reduce tail risks (e.g., missed unsafe cases) that have outsized legal or reputational costs, affecting insurance and capital allocation decisions.
• Investment and market competition
  • Firms that successfully embed process‑aware multi‑agent systems gain a competitive advantage in high‑trust applications; this may accelerate investment in domain‑specific models and tooling for process mapping and monitoring.
  • Standardization around such frameworks could lower entry costs for smaller providers to serve regulated clients, but also raises incumbent advantages for those who own labeled data and tuned escalation policies.
• Measurement and policy
  • New metrics (agent‑level uncertainty, transition dynamics, cost of escalation) introduce richer ways to evaluate GenAI deployment economics beyond raw accuracy — enabling finer cost‑benefit analyses for automation thresholds.
  • Policymakers and regulators can use auditable MDP traces to set compliance standards or certification procedures, which could shape market structure (e.g., certification premiums, required oversight staffing).
• Cautions for economic modeling
  • Empirical generalization is uncertain: small‑sample experimental evidence and model dependence mean realized labor and error reductions could differ across domains and institutions.
  • Strategic interactions: if many firms adopt similar multi‑agent workflows, equilibrium effects (e.g., wage compression for basic reviewers, premium for oversight skills) need to be monitored.

Overall, the paper provides a practically oriented, quantifiable architecture for mapping enterprise process maps to multi‑agent GenAI systems. For economists, it highlights a concrete mechanism by which AI can reallocate labor, change cost structures in regulated services, and create new measurement dimensions for policy and investment decisions.