A live benchmark shows state-of-the-art LLM agents complete at most two-thirds of realistic workflow tasks: the top model passes 66.7% of 105 controlled tasks. Failures cluster in HR, management and multi-system business workflows, indicating end-to-end workflow automation remains far from solved.

Main Finding

Claw-Eval-Live is a time-stamped, refreshable benchmark for end-to-end workflow agents that (1) sources task mixtures from public workflow demand signals (ClawHub Top‑500) and (2) grades agents by verifiable execution evidence (traces, audit logs, post-run artifacts) rather than final-text plausibility alone. In the current public snapshot (105 executable tasks spanning service-backed business workflows and local workspace repair) 13 frontier models were evaluated and the best model passed only 66.7% of tasks—showing that reliable workflow automation is far from solved and that failures cluster by task family and execution surface.

Key Points

• Design principle: “grounded twice”
  • Ground 1: Task mix is calibrated to fresh external demand signals (ClawHub Top‑500) so releases track evolving workflows.
  • Ground 2: Task success is anchored in observable execution evidence (tool traces, service audit logs, workspace state), with structured LLM judging only for semantic aspects not covered by deterministic checks.
• Release snapshot
  • Current public snapshot: 105 tasks, 22 fine-grained families, 18 controlled services, 13 public models evaluated.
  • Execution surfaces: 87 service-backed workflows (CRM, finance, email, calendar, helpdesk, KBs, multi-system coordination) and 18 workspace-repair tasks (terminal, file edits, tests).
• Construction pipeline (signal → snapshot)
  • Stage 1: Collect time-stamped ClawHub Top‑500 skill signals.
  • Stage 2: Cluster signals into workflow patterns preserving execution-relevant distinctions.
  • Stage 3: Convert patterns into family weights (distributional prior).
  • Stage 4: Seed expansion and implementation into executable candidate tasks; pilot screening for runnability and discrimination.
  • Stage 5: Discrimination-aware public selection: from 157 screened candidates, choose a public subset (N) using a MILP that balances release size, family coverage, and preservation of pilot-model ordering while excluding zero-discrimination tasks.
• Grading methodology
  • Primary evidence sources: data retrieval (tool discipline), data accuracy (entities/numbers vs ground-truth), and action verification (state-changing writes, workspace fixes).
  • Hybrid graders: deterministic checks first; structured LLM judges only for constrained semantic/organizational criteria.
  • Typical run settings: default budget = 24 turns, 300 seconds (longer when necessary for repair tasks); no model-specific prompt tuning.
• Empirical findings
  • Best model pass rate = 66.7%; no model reached 70%.
  • Service-backed, multi-system business workflows (HR, management, cross‑system coordination) remain harder than local workspace repair.
  • Leaderboard rank (pass rate) alone is insufficient: models with similar pass rates can diverge in completion patterns; most discrimination occurs in a middle band of tasks.

Data & Methods

• Signal source: ClawHub Top‑500 snapshot (external, time-stamped).
• Candidate pool: 157 screened runnable candidate tasks after pilot screening.
• Public release selection: mixed-integer linear program (MILP)
  • Binary selection variables xt ∈ {0,1} for each task t.
  • Objective maximizes preservation of pilot-model ordering p(i,j)t across top-K pilot models.
  • Constraints: fixed release size N, at least one task per fine-grained family Cf, exclusion of zero-discrimination tasks.
• Execution environment: each task packaged with YAML task definition, fixtures, tool schemas (RESTful controlled services), and task-specific grader (grader.py). Controlled services record audit logs; workspaces are sandboxed and preserve post-run artifacts.
• Grading
  • Deterministic checks where possible (tool call logs, audit trails, exact-match of entities/numbers, post-run tests).
  • Structured LLM judging only when required, constrained by explicit rubrics.
  • Three recurring grader archetypes:
  • Evidence-plus-judge for analytical tasks (deterministic checks + semantic rubric judge).
  • Operation verification for service workflows (heavy emphasis on audit trail checks).
  • Workspace repair verification (command traces + post-run artifact/tests).
• Evaluation protocol: same prompts, tools, fixtures for all models; runs recorded (tool calls, tokens, wall time, artifacts); per-task score in [0,1].

Implications for AI Economics

• Benchmarking that tracks demand: Using public signal priors (ClawHub) aligns evaluation mixes with what users currently try to automate—important for estimating real-world economic impact of agent automation (which workflows are most valuable/urgent).
• Productivity vs reliability trade-offs: Reported pass rates (best = 66.7%) imply substantial remaining friction. Economic adoption of agents will depend not only on average task completion but on verifiable correctness for high-value workflows (especially HR, approvals, cross‑system actions), affecting firms’ willingness to substitute human labor or reassign tasks.
• Deployment & contracting: Action-grounded evidence (audit logs, post-run artifacts) enables enforceable SLAs, liability allocation, and measurable performance-based pricing—critical for enterprise procurement and commercial contracts for agent-as-a-service.
• Sectoral heterogeneity in automation potential: The finding that HR, management, and multi-system coordination are harder suggests uneven automation substitutability across job tasks; simpler, localized repair/technical tasks may see earlier automation-driven productivity gains than complex socio-technical workflows.
• Investment signals: Firms and model providers may prioritize improving cross-system state manipulation, reliable writes, and action verification (engineering effort and R&D), rather than only improving fluent text generation—this steers capital allocation in AI tooling and integration services.
• Benchmark refresh cadence matters for economic measurement: A static task mix becomes stale relative to shifting enterprise needs; time-stamped, refreshable benchmarks permit more accurate tracking of model progress against evolving market demand—important for ex ante ROI forecasts and policy analyses.
• Risk and compliance externalities: The benchmark’s emphasis on auditable execution highlights regulatory and reputational risks from silent failures (plausible outputs without correct actions). Economically, this increases the value of trustworthy logging, verification tooling, and insurance products.
• Procurement and specialization: Since general leaderboard rank is insufficient, buyers should evaluate agents on a task-family basis (or require domain-specific benchmarks). This favors niche/specialized agent stacks and integration services over monolithic general-purpose models for enterprise deployments.

Summary: Claw-Eval-Live provides a principled, reproducible way to measure how well agents actually "do the work" demanded in the field, not just whether they write plausible reports. For AI economics, it sharpens measurement of where automation value and risk lie, which in turn affects firm adoption, contracting, investment priorities, and sectoral labor impacts.