A new 140‑hour enterprise meeting benchmark shows a learned dual‑policy assistant that routes queries between fast and slow reasoning and invokes retrieval/tools delivers better factuality‑latency tradeoffs than single‑model baselines and commercial systems; the approach is validated with expert‑grounded question protocols, ablations, and a real deployment case study.

Main Finding

MeetBench-XL introduces a realistic, enterprise-grounded meeting benchmark (MeetAll + MeetBench-XL) and a learned dual-policy meeting agent (MeetMaster-XL) that jointly optimizes routing and tool use to deliver substantially better quality–latency–cost trade-offs than single-model baselines or naive RAG. Key empirical outcomes: a validated 140-hour bilingual meeting corpus with 1,180 injected queries (κ = 0.82; injected turns are hard to detect, 56% discriminability), an automated evaluator calibrated to humans (Pearson r = 0.78), and a lightweight learned router (≈300K params, 1 ms overhead) giving 81.2% routing accuracy, ~30% latency reduction on simple queries, and a 12.4% absolute quality gain on complex queries — all on hardware deployable as a single RTX 4090 (22 GB).

Key Points

• Dataset & realism
  • MeetAll: 231 meetings (140 hours), bilingual (70.6% Mandarin / 29.4% English), 1,180 injected enterprise-informed queries across 13 balanced complexity classes.
  • Injection protocol anchored to stakeholder interviews (n = 15) and expert validation (κ = 0.82). Injected queries are near-natural (56% detection ≈ random).
• Multi-dimensional benchmark
  • MeetBench-XL evaluates five enterprise-prioritized dimensions: factual fidelity, user intent alignment, response efficiency (conciseness), structural clarity, completeness.
  • Automated LLM-as-judge calibrated via isotonic regression to human ratings (r = 0.78), enabling scalable evaluation.
• Agent architecture and performance
  • MeetMaster-XL: dual-process (fast Talker vs. slow Planner), learned joint policy for routing + tool invocation trained offline on quality–latency–cost objectives.
  • Router: small classifier (≈300K params, 1 ms overhead) improves routing accuracy to 81.2% (from 70.4% baseline).
  • Gains: 30% latency reduction for simple queries (P50 ≈ 0.92s → lower), 12.4% quality increase on complex queries (overall score 6.59 vs. 6.36), TE-Success@1 = 0.62 vs. 0.52–0.58 for baselines.
  • Deployable on consumer GPU (RTX 4090, 22 GB) with modest hardware cost (authors cite $1.6K vs. $30K+ for heavier setups).
• Resource and robustness numbers
  • Latency targets and measurements: simple queries P50/P90 ≤1.5s/2.0s (reported P50 0.92s, P90 1.35s on A100); complex queries P50/P90 up to 20s/30s (reported ~18.9s/26.7s).
  • Token costs: Talker avg ~0.4k tokens, Planner ~1.8–2.0k tokens per turn (varies by GPU and concurrency).
  • Robust to noise and multilingual split: quality degrades gracefully (clean 6.59 → SNR 10 dB 6.45 → 0 dB 6.20), language/domain rankings stable.
• Benchmarked vs commercial APIs / baselines
  • MeetMaster-XL competitive with closed APIs (GPT-4o, Claude 3.5, Gemini 1.5) while operating at substantially lower cost (paper reports an ∼18× lower cost comparison for their setting).
• Ablations & execution modes
  • Routing-only, tools-only, serial vs parallel execution studied. Parallel Talker+Planner reduces P90 latency significantly without harming quality.

Data & Methods

• Data construction
  • Seed corpora: AISHELL-4 (Mandarin, 211 meetings, 120h) and CHiME-6 (English, 20 meetings, 40h).
  • Injection cadence: roughly every 5 minutes per meeting; questions generated with LLM prompts then validated and human-refined by annotators (5 experts used to validate 200 questions).
  • Taxonomy: four orthogonal axes — Cognitive Load (CL), Context Dependency (CD), Domain Knowledge (DK), Task Execution Effort (TE); consolidated 108 cells into 13 classes to match enterprise frequency and ensure sample sizes.
  • Ground truth: candidates generated by strong LLMs then human-edited; human-finalized references preferred over raw LLM outputs in 91% of blind re-scorings.
• Evaluation methodology
  • MeetBench-XL: five 1–10 dimensions aggregated (equally weighted) and isotonic-calibrated to human judgments on a development set (calibration yields Pearson r = 0.78 on held-out validation).
  • Human annotation: four annotators, Cohen’s κ = 0.87 on rubric; sensitivity analyses show stable rankings.
• Agent design and training
  • Dual-policy flow: a small classifier routes queries to Talker (fast, low-cost, cached) or Planner (slow, tool-augmented, multi-step).
  • Planner executes a planning loop with tool invocation (retrieval, cross-meeting aggregation, web search), composes answers, and returns structured plan traces for auditability.
  • Training: offline policy learning / supervised fine-tuning for classification (focal loss) and RL-style optimization optimizing quality–latency–cost objectives (details in paper).
  • Execution: parallel Talker and Planner processing with token-level interrupts and Talker prefill to bound tail latency.
• Hardware & metrics
  • Experiments reported on A100 80GB (baseline), A40 48GB, and RTX 4090 (22GB) with throughput/tokens and latency numbers listed (e.g., RTX 4090 P50 latency ~1.23s (Talker) / 23.8s (Planner) depending on load).
  • TE metrics: TE-Success@1, ToolChain Length, Execution Time.

Implications for AI Economics

• Cost-efficiency and procurement
  • Learnable routing + selective tool use materially reduces compute and latency compared with naive universal RAG. That can lower per-query compute and token costs, enabling viable on-premise deployment on modest GPUs (RTX 4090) rather than expensive multi-A100 clusters—substantial capex/OPEX savings for enterprises.
  • Reported deployment claim: single 22 GB GPU (≈$1.6K) vs. multi-A100 setups (authors cite $30K+), plus reported 18× lower cost relative to some API-heavy baselines in their experiments. These numbers suggest large potential ROI for enterprises that process high query volumes and have privacy/latency constraints.
• Latency, abandonment, and productivity value
  • Empirical evidence: user abandonment spikes when latency >3s (34% abandonment in their deployment study). Reducing P50/P90 for simple checks to sub-1.5s via a fast path reduces abandonment risk, likely translating to measurable productivity gains (fewer re-asks, faster decisions).
  • Economists evaluating adoption should weigh marginal latency improvements against hardware and engineering costs; the dual-policy approach concentrates heavier compute only where it yields quality gains.
• Pricing, marginal-cost accounting, and capacity planning
  • The docs provide concrete per-turn token profiles (Talker ~0.4k, Planner ~1.8–2.0k) and latency distributions by device. These allow finance/ops teams to model per-query marginal cost under different traffic mixes (fraction simple vs complex) and estimate break-even points for on-premise vs cloud/API usage.
  • Because the router reduces unnecessary Planner invocations, the effective marginal cost curve is steeper for complex queries but flatter for the bulk of routine queries—affecting optimal procurement (smaller GPU pools with smart routing vs. large always-on clusters).
• Evaluation economics and scaling
  • Calibrated LLM-as-judge (r = 0.78) reduces human labeling load for large-scale benchmark runs; that lowers evaluation cost and speeds iteration, but calibration and maintenance are required to prevent drift—an ongoing operating expense.
• Privacy, compliance, and hidden costs
  • On-premise deployment reduces data-exfiltration risks and API fees but shifts responsibilities (software maintenance, security, compliance). Total cost-of-ownership should include engineering, model-update, and governance expenses.
• Investment and product strategy signals
  • For enterprises with heavy meeting workloads and privacy/latency needs, investing in a learned, resource-aware agent (lightweight router + conditional tool use) likely gives superior welfare (lower costs + higher quality). Vendors can differentiate by selling highly optimized on-premise solutions or hybrid models that selectively use cloud tools for the Planner tier.
• Caveats for economic modeling
  • Reported savings and QoL gains depend on query distribution (their taxonomy aligns to their partners’ usage). Firms with very different mixes (more high-complexity queries) will face higher marginal Planner costs and different ROI.
  • Dataset creation and annotation required significant human effort (stakeholder interviews, expert validation, human finalization). Those upfront costs matter when considering reproducibility and adoption at scale.

Suggested quantitative next steps for economic assessment - Use provided token and latency stats to compute per-query marginal cost under different traffic mixes (e.g., 70% low-CL → Talker vs 30% Planner). - Estimate productivity gains from reduced abandonment (use their 34% abandonment >3s figure as baseline) and monetize by average meeting-hour value. - Include recurring costs: model updates, calibration maintenance, and security/infra overhead for on-premise deployments.

If you want, I can: - Sketch a simple cost model (per-query cost and break-even for GPU purchase vs API spend) using your organization’s query volume and complexity mix. - Extract the key numeric tables (latency, token costs, quality scores) into a compact CSV-style summary for spreadsheet modeling.