1 cumulative citations
View corpus contextA new audio benchmark finds wide differences in voice-agent performance: some systems autonomously resolve 71% of simulated credit-card calls while others resolve just 43%, with most failures occurring during verification or execution despite high connect rates; the benchmark uniquely cross-checks spoken claims against actual database actions.
VAmoS Bench: Voice Agent Simulation Bench
VAmoS Bench is an audio-first, stateful simulation benchmark of 100 credit-card support scenarios that measures end-to-end containment (task completion) of voice-agent systems by jointly grading conversation transcripts and execution traces, finding completion rates from 43% to 71% across 11 stacks.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Production voice agents span cascaded, speech-to-speech, and hybrid architectures. Voice-agent benchmarks typically measure component quality and conversational properties such as word error rate, latency, naturalness, and turn-taking. Fewer measure whether the agent handled a phone call correctly on its own. Contact centers refer to this as ``containment'': the share of phone calls the automated system resolves without handing off to a human. On some phone calls the right outcome is refusal or a redirect. To address this gap, we introduce VAmoS Bench, the Voice Agent Simulation Bench. It measures complete voice-agent systems end to end in a stateful customer-support task. The agent is Riley, a credit-card support representative for a fictional bank who can freeze, cancel, replace, or activate a card. Each of 100 scenarios supplies a simulated caller with a private goal and a seeded PostgreSQL backend. The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio; roughly one-third apply adversarial pressure. The agent can use five tools that execute real SQL against the backend. Each scenario also defines binary assertions. A grader evaluates them against the complete trace of what the caller and agent said and what the agent did, including tool invocations, arguments, and returned rows. This catches an agent that claims to have changed a card without updating the database, as well as one that makes the right database change while disclosing protected information. This first benchmark version focuses on financial services. Its evaluation protocol supports an evolving leaderboard: additional voice agents can be evaluated on the same version, while later versions can expand the tasks and scenarios.
Summary
Main Finding
VAmoS Bench introduces an end-to-end, stateful benchmark for voice agents that judges “containment” — whether an automated voice system correctly handles a full phone call without human handoff — by evaluating the combined audio transcript and the execution trace (tool calls + database state). Applied to 11 voice-agent stacks on a 100-scenario credit-card support task, the benchmark reveals wide variation in containment (43.0%–71.0%), latency, and estimated per-call cost, and shows many failures happen after connection (during verification, tool use, or privacy-sensitive steps), not in basic connectivity.
Key Points
- Purpose: Measure real-world deployment value (resolved phone calls / containment) rather than isolated component scores (WER, TTS quality).
- Core idea: Jointly evaluate what the agent said and what it actually executed by comparing the conversation transcript, tool invocations, and returned database rows against fixed scenario assertions.
- Task: “Riley” card-operations agent (freeze, cancel, replace, activate cards; status & delivery checks) interacting via live audio with a simulated caller.
- Scenario set: 100 isolated phone-call simulations, each with a seeded PostgreSQL backend, a caller persona and private goal; 32/100 are adversarial/guardrail scenarios designed to pressure verification and confidentiality.
- Tools: Five real SQL-backed tools (lookup user, find card by last-4, change card status, request replacement, update delivery). Tool invocations execute real SQL against per-simulation Postgres instances.
- Verification policy: Riley must verify last 4 digits + name + address or phone before sharing anything or changing state — assertions can require verification-before-action and forbid disclosure prior to verification.
- Grading: An LLM judge applies scenario-specific natural-language assertions to the combined trace (transcript + tool calls + returned rows). Agent PASS only if all assertions pass for a scenario.
- Operational metrics recorded: connect rate, median/mean/p90 response latency (actor-side), median phone-call duration, turns, barge-in (interruptions per 100 calls), and estimated per-call cost.
- Scope and runs: 11 voice agents, each evaluated with 3 full runs (100 sims/run) → 3,300 phone calls. 3,243 phone calls returned verdicts; 57 missing verdicts (55 never connected; 2 with no gradeable caller turn).
Data & Methods
- Simulation environment:
- Each scenario runs in an isolated container with a seeded Postgres DB (per-call isolation).
- Caller: simulated, interactive, holds private goal across turns, can re-assert or escalate; accents and speaking characteristics are parameterized.
- Audio transport: WebSocket carrying PCM16 @ 24 kHz; adapters support WebRTC, webhook, and Twilio for telephony evaluations.
- Tool dispatch: flexible — in-process, WebSocket, or HTTPS webhook via per-simulation tunnel. Tool semantics and backend are identical for all agents.
- Scenario design:
- Generated from agent policy, tool schemas, and task description; manually reviewed.
- 100 scenarios: 30 simple flows, 38 complex flows, 32 adversarial/guardrail (mutually exclusive partition for analysis).
- Assertions: 81 scenarios have one assertion, 17 have two, 2 have three; assertions are fixed and identical across evaluated agents.
- Agents evaluated:
- Eleven stacks across categories (self-hosted orchestration, hosted platforms, bundled APIs, native speech-to-speech endpoints). Examples: Pipecat, LiveKit Agents, Vapi, ElevenLabs ConvAI, Cartesia, OpenAI Realtime (full & mini), Gemini 2.5 / 3.1, Retell, Nemotron (NVIDIA models).
- Component choices: where frameworks exposed components, authors pinned ASR, LLM, and TTS to allow closer comparisons (e.g., Deepgram + gpt-4.1-mini + ElevenLabs TTS for several stacks).
- Grading protocol:
- Judge uses the full trace (spoken turns, tool invocation sequence & args, returned rows) and applies scenario assertions to produce PASS/FAIL and justification per assertion.
- Task completion per agent = share of phone calls where all assertions pass.
- Runs & statistics:
- Each agent: 3 runs × 100 scenarios → 300 calls; pooled metrics over all calls; completion error bars = standard error across the three run-level completion rates (not full CIs).
- Results highlights (pooled over calls):
- Overall containment across agents: 63.1% (2,081/3,300).
- Best observed completion: Pipecat 71.0%; LiveKit Agents 70.3%; worst: Nemotron 43.0%.
- Connection rate high overall: 98.3% connected (3,245/3,300); most failures occurred post-connection.
- Median response latency (agent-side) varied: Pipecat 1.95s; ElevenLabs ConvAI 1.19s (lowest); Gemini 2.5 had a large median latency (15.95s) in this study.
- Median call durations spanned ~73s to ~143s.
- Median turns per call typically 7–13; barge-in (agent interruptions) varied widely (Retell had 126 interruptions per 100 calls).
- Estimated per-call cost (modeled/metred): ranged from about $0.016–$0.208 in table; Pipecat ~$0.045, LiveKit ~$0.048, Retell ~$0.208. (Nemotron cost not reported.)
- Limitations the authors note:
- Judge reliability and bias (LLM judge); synthetic caller fidelity; domain limited to financial card operations; scenario set size and composition; runs limited to three repetitions per agent; 1st-version benchmark focused on financial services.
Implications for AI Economics
- Containment (task completion) is the economic unit of value for enterprise voice agents:
- Buyers should value benchmarks that measure resolved calls end-to-end (containment), not only component scores. Containment directly relates to labor substitution, operational savings, and SLA performance.
- Procurement and vendor choice:
- Procurement decisions should weigh containment, latency, and per-call cost jointly. A lower per-minute or per-token price may not yield better economics if containment is low or if privacy failures impose compliance/reputational costs.
- Benchmarks like VAmoS enable apples-to-apples comparisons across self-hosted frameworks, hosted platforms, and bundled APIs, reducing information asymmetry between vendors and buyers.
- Pricing, SLAs, and contracting:
- Sellers may be pushed to offer SLAs tied to containment and safety metrics (e.g., verification-before-disclosure) rather than only uptime or latency. Contracts should internalize both successful containment and failure modes that cause escalations or privacy breaches.
- Benchmarked containment can be used to estimate ROI of automation (hours saved, human-agent FTEs reduced) more reliably than component proxies.
- Competitive differentiation & innovation incentives:
- Because joint grading exposes path-dependent failures (e.g., claims without DB changes, disclosure before verification), vendors optimizing for end-to-end containment will invest differently (better tool orchestration, stricter guardrails, secure execution pipelines) than vendors optimizing only for ASR/LLM/TTS metrics.
- Lower-latency stacks with similar containment rates have a competitive edge (improved customer experience, lower call time), affecting pricing and market share.
- Labor-market impacts:
- Containment rates from realistic benchmarks inform more accurate estimates of job displacement or role changes in contact centers. A 70% containment implies different staffing and retraining needs than a deployment with 43% containment.
- Partial containment (correct refusal/redirects but with disclosure leaks) complicates simple substitution calculations—legal/complaint costs may offset labor savings.
- Externalities, risks, and regulation:
- Benchmarks that surface confidentiality breaches and adversarial failures can influence regulation and compliance requirements (e.g., explicit verification ordering). Regulators and auditors may demand evidence of end-to-end correctness, not just component audits.
- Adversarial scenarios show economic risk from malicious callers; vendors and buyers must price expected remediation, liability, and monitoring costs into deployment economics.
- Vendor lock-in and switching costs:
- Real-data tool integrations, verification logic, and per-call tooling become part of a deployment’s value; benchmarks that standardize tool interfaces can lower switching friction, but differences in contained performance and costs still create lock-in forces.
- Product-market structure:
- The market may bifurcate: high-containment, higher-cost offerings (enterprise-safe, low privacy risk) vs. lower-cost, lower-containment offerings for lower-stakes use cases. Benchmarks help segment offerings by demonstrated containment and safety.
- Measurement & market transparency:
- Standardized, public benchmarks that measure containment (and safety) can reduce information asymmetries and enable more efficient contracting, pricing, and competition among voice-agent providers.
- Investment decisions:
- Organizations choosing between self-hosting and vendor APIs can use containment × cost × latency tradeoffs to decide capital vs. operating expenditure: self-hosting may allow more control (and possibly lower per-call costs) but requires engineering to reach high containment and safety.
- Research & policy relevance:
- Economists and policymakers interested in automation impacts should prefer deployment-level measures (containment, escalation rates, privacy failures) when modeling labor impacts, rather than component-level proxies.
Overall, VAmoS Bench operationalizes an economically relevant outcome — resolved calls without human handoff and without privacy-policy violations — and provides a concrete instrument for buyers, vendors, and researchers to compare end-to-end performance, cost, and risk. For AI economics, the benchmark supports more accurate ROI calculations, procurement decisions, SLA design, and labor-impact estimates, while highlighting the importance of path-sensitive safety checks for valuing automation.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents a substantial empirical benchmark (3,300 phone calls across 11 voice-agent stacks, with isolated seeded backends and three repeated runs) and measures end-to-end task completion rather than component metrics. However, the evaluation relies on synthetic simulated callers and an LLM-based judge (known sources of bias and reliability concerns), covers a single domain (credit-card operations) with 100 curated scenarios, and is a snapshot of vendor/model versions, which limits causal claims and broader external validity.
Methods Rigormedium — Carefully designed simulation infrastructure (per-scenario isolated PostgreSQL backends, real SQL tool execution, transport adapters for multiple deployment types), repeated runs, and joint grading of conversation plus execution trace strengthen rigor. Limitations include reliance on an LLM judge and synthetic callers, a fixed 100-scenario set chosen/curated by authors, only three runs per agent, and limited discussion of inter-annotator or judge calibration/validation.
Sample11 voice-agent stacks evaluated via 100 distinct scenarios (30 simple, 38 complex, 32 adversarial/guardrail), each scenario instantiated in isolation with a seeded PostgreSQL backend and five real SQL-executing tools; simulated callers (with persona, private goal, accent/style) interacted over audio; each scenario run three times per agent for a total of 3,300 phone calls (3,243 produced verdicts); per-call traces include transcripts, tool invocations/arguments/returned rows; agents include cascaded and speech-to-speech stacks (Pipecat, LiveKit Agents, Vapi, ElevenLabs ConvAI, Cartesia, OpenAI Realtime variants, Gemini variants, Retell, Nemotron).
Themesproductivity adoption human_ai_collab
GeneralizabilityDomain-limited: scenarios focus on credit-card operations and may not generalize to other contact-center or voice-agent tasks., Synthetic callers: simulated user behavior may not capture the full variability and adversarial strategies of real customers., Judge reliability: use of an LLM judge introduces risks of inconsistent or biased PASS/FAIL decisions without independent human validation., Snapshot of tooling: vendors, model versions, and orchestration choices change rapidly so results are time-bound., Language/accent coverage unclear: results may not generalize across languages or diverse accents beyond the tested set., Limited runs and scenario set: 3 runs and 100 scenarios constrain statistical resolution and coverage of rare edge cases.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| VAmoS Bench evaluates complete voice-agent systems end to end on stateful customer-support phone calls rather than measuring only component-level metrics such as word error rate, latency, naturalness, or turn-taking. Organizational Efficiency | positive | End-to-end phone-call task completion |
Reading fidelity
high
Study strength
medium
|
n=100
|
| The first VAmoS benchmark version contains 100 scenarios involving a simulated caller, a private caller goal, and a scenario-seeded PostgreSQL backend. Other | positive | Benchmark scenario coverage |
Reading fidelity
high
Study strength
medium
|
n=100
100 scenarios
|
| VAmoS grades the conversation and execution trace jointly, using spoken content, tool invocations, tool arguments, and returned database rows as one evidence record. Regulatory Compliance | positive | Correctness and policy compliance of voice-agent actions and statements |
Reading fidelity
high
Study strength
medium
|
n=100
|
| Thirty-two of the 100 scenarios applied adversarial pressure or guardrails. Ai Safety And Ethics | positive | Exposure to adversarial and guardrail scenarios |
Reading fidelity
high
Study strength
high
|
n=100
32%
|
| Across the 3,300 phone calls, voice agents completed the benchmark task on 2,081 calls, corresponding to an overall completion rate of 63.1%. Organizational Efficiency | mixed | Share of phone calls completed successfully |
Reading fidelity
high
Study strength
medium
|
n=3300
63.1% completion; 2,081 completed phone calls
|
| Observed task completion varied from 43.0% for Nemotron to 71.0% for Pipecat across the evaluated voice-agent stacks. Organizational Efficiency | mixed | Voice-agent benchmark completion rate |
Reading fidelity
high
Study strength
low
|
n=3300
43.0% to 71.0%
|
| Connection failure was not the main source of failure: 98.3% of phone calls connected, while most failures occurred after connection during the conversation or execution of the requested task. Error Rate | negative | Post-connection task failure and call connection rate |
Reading fidelity
high
Study strength
medium
|
n=3300
98.3% connected
|
| The reported completion rankings should be interpreted as a snapshot of observed performance rather than a definitive ranking. Organizational Efficiency | mixed | Stability and comparative precision of completion rates |
Reading fidelity
high
Study strength
high
|
n=3
0.9 to 5.5 percentage points
|
| Pipecat achieved the highest observed completion rate among the 11 evaluated agents, at 71.0%, with a median response latency of 1.95 seconds and estimated cost of $0.045 per phone call. Organizational Efficiency | positive | Task completion rate |
Reading fidelity
high
Study strength
low
|
n=300
71.0% completion
|
| Gemini 2.5 Native Audio had the highest median response latency in the reported comparison, at 15.95 seconds. Task Completion Time | negative | Median response latency |
Reading fidelity
high
Study strength
medium
|
n=300
15.95 s median latency
|