An always-on intermediary that routes scarce human attention to ongoing agents raises task-quality scores substantially across diverse LLM workers in a new benchmark, with improvements of roughly 5–28 points on a 0–100 scale. The gains vary by both intermediary and worker model and were measured using simulated users and a curated suite of tasks, so real-world benefits and human-cost tradeoffs remain to be validated.

Main Finding

An always-on intermediary layer (Jarvis) that separately coordinates scarce human attention with ongoing agent work meaningfully improves task outcomes and user-facing interaction across diverse agent runtimes. The authors introduce JarvisBench—a 45-task benchmark (20 single-agent, 25 multi-agent) and two evaluation tracks—that quantifies bidirectional attention coordination, showing task-score gains of roughly 4.9–24.7 points (single-agent) and 12.5–28.2 points (multi-agent) depending on worker/Jarvis LLM. GPT-5.6-Sol was the strongest Jarvis in these experiments (Overall user-interaction score 96.3).

Key Points

• Problem framed: modern agents run continuously while human attention is intermittent; coordination must work both ways:
  • Agent→User: detect when human judgment is needed and solicit it timely.
  • User→Agent: provide an always-on interface so users can query/steer agents without interrupting execution.
• JarvisBench design:
  • 45 agentic task instances spanning 19 domains, selected/adapted from >2,000 candidates.
  • Tasks generate attention needs organically during execution (not by withholding prompt info).
  • Jarvis runs as an external sidecar observing bounded task events and using a narrow guidance interface—worker loops are unchanged.
• Two evaluation tracks:
  • Agent-Collaboration Track: does human attention, mediated by Jarvis, improve outcomes? (Outcome metric: 0–100 Task Outcome Score from weighted checkpoints.)
  • User-Interaction Track: is Jarvis useful when the user initiates contact? (Causal replay protocol with early/late checkpoints; answers graded 0/1/2, converted to 0–100.)
• Intervention protocol used in experiments: when Jarvis intervenes it pauses the worker at an action boundary, cancels the pending action, asks a focused user question, and injects scoped “soft guidance” before resuming—balancing outcome gains vs. attention/disruption.
• Empirical summary:
  • Jarvis (GPT-5.6-Sol) improved every evaluated worker configuration.
  • Single-agent gains ranged ~4.9–24.7 points; multi-agent gains ~12.5–28.2 points.
  • Attention Efficiency metric: (gain in score) / (100 × number of attention requests) quantifies score gain per requested turn (examples in paper show efficiencies ~0.09–0.41 depending on model/task).
  • GPT-5.6-Sol produced the largest absolute gains and highest user-interaction quality; some smaller models (e.g., DeepSeek V4-Pro) achieved higher attention efficiency in certain setups.
• Prototype also includes a full-duplex speech interface (Qwen3-ASR, Silero VAD, Kokoro-82M TTS) to demonstrate always-on spoken interaction and latency measurements.

Data & Methods

• Benchmark contents:
  • 20 single-agent, 10 multi-agent projects (25 multi-agent workstreams total), 19 domains, >2K candidates screened.
  • Multi-agent projects require coordination across coupled workstreams; project-level decisions emerge during execution.
• Roles & simulation:
  • Three separated roles: user (simulated by an LLM with a frozen task-specific user profile for controlled eval), Jarvis (intermediary LLM), and one or more working agents (unchanged runtimes).
• Metrics:
  • Task Outcome Score Si ∈ [0,100]: weighted sum of frozen checkpoints (objective execution, alignment, deliverable quality, safety).
  • Agent-Collaboration outputs: baseline score (no Jarvis), Jarvis score, mean number of attention requests, and Attention Efficiency = (S̄Jarvis − S̄base) / (100 × N̄req).
  • User-Interaction: causal replay of recorded worker trajectories with fixed General and model-generated Follow-up questions at ~25% and ~75% execution points. Responses scored 0/1/2, converted to 0–100.
• Experimental setup:
  • Two main comparisons: 1) Fix Jarvis = GPT-5.6-Sol; vary worker models (Claude Opus 5.0/4.8, GPT-5.6-Sol, GPT-5.5, DeepSeek V4-Pro, GLM 5.2). 2) Fix worker = Claude Opus 4.8; vary Jarvis LLM (GPT-5.6-Sol, Claude Opus 4.8, DeepSeek V4-Pro, GPT-OSS-120B).
  • User-Interaction track compared GPT-5.6-Sol, Claude Opus 4.8, DeepSeek V4-Pro, Qwen235B, GPT-OSS-120B. Latency measured as time to first audio (deployment-dependent).
• Implementation notes:
  • Jarvis is attached via bounded task events and a narrow guidance API; it does not use worker tools or alter worker planning/loop.
  • Prototype voice stack components and turn controller described; full-duplex interaction supported.

Implications for AI Economics

• Human attention as a scarce economic input:
  • JarvisBench quantifies returns to human attention via Attention Efficiency (score gain per requested turn). That creates a measurable relationship between marginal human supervisory input and marginal outcome improvement—useful for costing and pricing human-in-the-loop services.
• Valuation and productization of intermediary services:
  • External attention-coordinators (Jarvis-like sidecars) can be a distinct product category: platforms can monetize always-on mediation that improves multi-agent outcomes without changing agent internals.
• Labor allocation and roles:
  • Results imply a shift in human labor from doing tasks to being intermittent supervisors/decision-makers; demand increases for lightweight supervisory capacity (higher-value, lower-volume attention) rather than continuous task execution.
  • Pricing models could be per-attention-turn, subscription for always-on availability, or performance-based (pay for task outcome improvements); Attention Efficiency helps calibrate these prices.
• Platform and vendor competition:
  • Choice of Jarvis LLM substantially affects ROI. Providers can compete on the intermediary quality dimension separately from agent capability—leading to layered markets (worker models vs. attention orchestration models).
• Cost-benefit and optimization tradeoffs:
  • The intervention strength tradeoff (more requests → higher potential quality but higher attention cost and disruption) creates a frontier for optimal policies. Firms will need to set operating points based on value of improved outcomes vs. human attention cost and latency tolerances.
• Welfare, safety, and regulation:
  • For safety-critical applications, separating coordination from execution enables standardized oversight layers; regulators might mandate an always-on attention coordinator with audited intervention logs.
  • However, over-requesting or poorly calibrated intervention policies can create cognitive load and negative externalities—market mechanisms or regulation may be needed to prevent excessive attention extraction.
• Measurement and evaluation economics:
  • JarvisBench provides a stable evaluation target across agent upgrades (because it sits outside worker loops), enabling apples-to-apples comparison of intermediary designs and facilitating investment decisions based on measured gains per attention-turn.

If you want, I can (a) extract the key quantitative results into a compact table for economic modeling (attention efficiency vs. worker/Jarvis pairs), (b) sketch a simple cost-benefit model using Attention Efficiency to price human attention, or (c) highlight use cases (e.g., software engineering, content moderation, design workflows) where intermediary coordination yields the largest economic gains. Which would be most useful?