Frontier agents have become far more capable and safer: task completion on WorkBench rose from 43% (GPT-4, Mar 2024) to 89% (Claude Opus 4.8, Jun 2026) and unintended harmful actions fell from 26% to 2.5%. At the same time, the rise of open-weight models has sharply reduced the cost of reaching top-tier performance, though rare irreversible errors such as mis-sending emails persist.

Main Finding

WorkBench, a 690-task sandbox benchmark for workplace agents, shows large improvements between 2024 and 2026: frontier task completion roughly doubled and unintended harmful side effects fell sharply. The best 2026 agent (Claude Opus 4.8) completes 88.8% of tasks with a 2.5% harmful-action rate. At the same time, open-weight models have driven dramatic cost reductions for a given capability level, while frontier (highest-capability) costs remain comparatively stable.

Key Points

• Overall performance
  • Best completion: 2024 frontier (GPT-4) → 43% (original reported) vs 2026 frontier (Claude Opus 4.8) → 88.8% on the corrected benchmark.
  • Across 21 evaluated models (Mar 2023–May 2026), completion ranges from ~26% to ~89%.
• Safety / harmful side effects
  • Harmful-action rates fell from ~26% (GPT-4, 2024 ReAct run) to as low as 2.5% (Opus 4.8); models that complete more tasks also tend to have fewer harmful side effects (capability and safety rise together on WorkBench).
• Cost
  • Cost-per-benchmark-run estimated from token use (upper-bound; no caching). Cost-per-task spans ~two orders of magnitude.
  • Examples: Opus 4.8 ≈ $0.182/task; GPT-5.5 ≈ $0.206/task; open-weight Qwen3.5 ≈ $0.003/task (≈100× cheaper than earlier GPT-4 baseline for a similar capability point).
  • The efficient frontier is mixed: cheap end dominated by open-weight (mostly Chinese labs), high-capability end dominated by Western proprietary models.
• Changes in benchmark / harness and measurement
  • New harness uses native structured tool-calling (avoids the ReAct format-adherence failures common in 2024).
  • Corrections to scoring, prompts, ground truth, and tools: a non-trivial share of tasks were fixed (8–14% of tasks affected).
  • Rescoring example: GPT-4’s score moved +8 percentage points (49% → 57%) on the corrected benchmark, so cross-year comparisons must use the same benchmark revision.
• Residual failure modes
  • Some classes of error eliminated (ReAct parsing failures, wrong-email caused by docstring imitation, updating wrong event).
  • Remaining problems include: mis-evaluating conditionals (acting when condition false), comparing incompatible metrics (percent vs raw value), trusting truncated search results (not re-querying when results cap applies), plotting or reasoning about future dates beyond sandbox data, and occasional irreversible harms (e.g., sending email to wrong recipient).
• Limits and caveats
  • Sandbox simplifications (smaller inbox/calendar than real workplaces) mean measured completion is optimistic relative to real-world deployment.
  • Cost estimates are upper bounds; providers caching system prompts/schema would pay less.
  • WorkBench is public; models trained after 2024 may have seen the benchmark in pretraining (no private hold-out), so observed improvements are an upper bound on genuine capability gains.

Data & Methods

• Benchmark
  • WorkBench: sandbox with five domains (calendar 300 events, inbox 500 emails, analytics 500 records, CRM 200 customers, PM board 300 tasks), 26 tools, and 690 templated tasks (10 per template).
  • Outcome-centric scoring: final sandbox state compared against ground truth; agent paths and recovery allowed.
• Experimental harness
  • Switched from ReAct (free-form tool-call parsing) to native structured tool-calling (models emit typed calls matching tool schema). Loop otherwise same: up to 20 steps, temperature=0 where available.
• Models evaluated
  • 21 models (March 2023–May 2026) including GPT line (3.5 → 5.5), Anthropic Claudes (Opus 4.8 Sonnet 4.6 Haiku 4.5), Google Geminis, several open-weight Chinese models (Qwen, DeepSeek, Kimi, GLM).
• Cost estimation
  • Token counts approximated (4 characters/token) + fixed per-call overhead (~7,000 tokens for system prompt and tool schema sent on every call).
  • Priced at providers’ published per-token rates without caching → conservative upper bound on cost.
• Benchmark corrections & scope
  • Fixes include off-by-one date cutoffs, prompt/answer-key mismatches, aggregation bugs, tool docstring clarifications, search result caps raised where necessary.
  • Deterministic ground-truth changes impacted 56 tasks (8%); including other fixes raises affected tasks to ~14%.

Implications for AI Economics

• Unit economics of agentization have shifted
  • Dramatic cost reductions for capable agents (open-weight models providing comparable or better capability at ~1–2 orders magnitude lower cost) lower the marginal cost of deploying agentic automation for many tasks.
  • Lower per-task compute costs reduce the threshold for commercializing automated workplace assistants and increase ROI on agent-enabled product features.
• Market structure and competition
  • Two-sided frontier: cheap, capable open-weight models (largely Chinese labs) compete at the low-cost, mid-capability range; Western proprietary models still often lead in absolute capability but at higher cost.
  • Western vendors’ mid/low-cost tiers face pressure from cheaper open-weight options that dominate on cost and sometimes capability—this may compress pricing and product tiering.
• Diffusion and labor effects
  • Better and cheaper agents accelerate substitution/augmentation of routine workplace tasks—firms face faster productivity gains but also faster labor-displacement risks in administrative roles.
  • Because frontier models still make rare irreversible mistakes, adoption strategies will favor human-in-the-loop patterns for high-risk tasks, slowing full automation for safety-critical operations.
• Investment and R&D incentives
  • Open-weight models changing cost dynamics may re-orient capital toward systems/integration, safety tooling, domain-specific fine-tuning, and operations (e.g., caching, orchestration) rather than raw model training.
  • Providers may differentiate on reliability, external guarantees, compliance, and integration rather than raw capability alone.
• Pricing, procurement, and regulation
  • Procurement should account for operational costs beyond per-token prices (caching, orchestration, error remediation), and for externalities from harmful side effects (liability costs).
  • Regulators and firms should monitor residual irreversible harms (wrong emails, data leakage) even as average harmful-action rates fall—risk remains tail-heavy.
• Measurement & policy recommendations
  • For robust economic estimates, future benchmarking should include:
    • A private held-out split (to avoid memorization effects) and refreshed annual held-out tasks.
    • Caching-aware cost accounting and billed-spend traceability.
    • Larger, more realistic sandboxes (scale of inbox/calendar) to better estimate deployed reliability and error remediation costs.
  • Policymakers should track both average cost-per-task and tail risk (frequency and magnitude of irreversible harms) when assessing systemic impacts.

Overall, WorkBench Revisited documents substantial capability and safety gains and large cost reductions for many agent-capable models between 2024 and 2026. The economic consequence is a lower marginal cost for building and deploying workplace agents—accelerating adoption—while leaving important open questions about residual risks, market concentration at the capability frontier, and real-world reliability costs that matter for firms and regulators.