0 cumulative citations
View corpus contextA unified deployment-aware benchmark finds no one LLM wins everywhere: model strengths vary by input modality and execution form, and accuracy–cost trade-offs differ across text and multimodal workloads; local open-weight models are competitive in pockets but generally trail frontier commercial APIs.
DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents
DAREBench presents a deployment-aware benchmark of 233 agentic tasks run in a unified execution environment with evidence-based auditing, showing workload-dependent model rankings and distinct accuracy–cost trade-offs where no single model dominates all workloads.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to assess multimodal perception, multi-step execution, tool use, and artifact delivery. However, existing benchmarks are often tied to specific task types, execution environments, or scoring protocols, limiting their comparability, interpretability, and reliability for deployment decisions. We introduce DAREBench (Deployment-Aware and Reliable Evaluation of Models as Agents), a benchmark designed to capture workload variation and support reliable agent evaluation. Built on a shared OpenClaw execution environment, DAREBench organizes 233 tasks selected and adapted from 22 source benchmarks into a $2\times3$ workload matrix defined by input modality and execution form, and evaluates them under a unified contract-based protocol with evidence-based score auditing. We evaluate 23 commercial API models and 12 locally deployed open-weight models over 7,587 model--task runs, reporting accuracy and token consumption alongside reference costs for API models. Results show that no single model dominates all workload groups, text and multimodal tasks exhibit distinct accuracy--cost trade-offs, and local open-weight models are competitive in several groups but still trail frontier commercial models overall. These findings suggest that agent deployment and model selection should consider workload profiles, deployment mode, and accuracy--cost trade-offs rather than rely on a single aggregate score.
Summary
Main Finding
DAREBench is a deployment-aware benchmark for models-as-agents that (1) organizes 233 curated tasks into a 2×3 workload matrix (text vs. multimodal × single-step, multi-step, multi-step+tools), (2) runs all models in a shared OpenClaw execution environment with contract-based tasks, and (3) applies evidence-based auditing (meta-judges plus deterministic checks) to reduce unsupported positive scores. Evaluating 35 models (23 commercial API and 12 local open-weight) across 7,587 model–task runs, the authors find no single model dominates all workloads; text and multimodal workloads exhibit distinct accuracy–cost trade-offs; locally hosted open-weight models are competitive on some workload groups but lag frontier commercial APIs overall; and auditing removed a nontrivial number (143) of unsupported positive scores.
Key Points
- Workload design
- 233 tasks adapted from 22 source benchmarks using a three-stage curation: feasibility (artifact contract definable), difficulty (exclude trivially solved), and representativeness (cover source categories).
- Tasks grouped into 6 workload groups: {text, multimodal} × {single-step, multi-step, multi-step+tools}.
- Unified execution & attribution
- All models run under a common OpenClaw agent loop, identical tool interfaces, workspace abstraction, and logging to reduce runtime-compatibility confounds.
- Scoring & audit
- Three scoring modes per task: automated (deterministic), LLM-as-a-judge, or hybrid.
- An evidence-based audit applies deterministic trigger rules and a meta-judge to flag and remove unsupported positive judgments; audited scores set to zero when unsupported.
- Audit removed 143 unsupported positive scores in the reported experiments.
- Evaluation scale and outputs
- 35 models evaluated (23 API models across families like GPT/Claude/Gemini/etc., and 12 locally hosted open-weight models).
- 7,587 applicable model–task runs; reported metrics include audited accuracy (overall and per workload group), token usage, and reference cost estimates based on public API pricing (per-million-token rates from OpenRouter).
- Findings on cost-performance
- Accuracy–cost Pareto frontiers differ between text and multimodal workloads; several models form different frontiers by modality.
- No universal winner; model ranking depends strongly on workload group.
- Local models can be competitive in specific groups but generally trail frontier commercial APIs when evaluated with API reference costs.
- Deployment realism
- Local models were run with a practical single-GPU FP16 32K-context configuration (no quantization, no multi-GPU), reflecting plausible on-premise setups, but cost estimates for local models are treated as third-party references rather than measured TCO.
Data & Methods
- Task set
- 233 tasks curated from 22 public benchmarks (reasoning, long context, QA, code/shell, GUI, multimodal search, etc.).
- Each task formalized as a tuple: (instruction, initial workspace, tool subset, artifact contract, scorer, time budget).
- Workload matrix
- Two modality classes: text vs. multimodal.
- Three execution forms: single-step, multi-step, multi-step+tools (tool-mediated).
- Execution environment
- Shared OpenClaw runtime used for all experiments.
- Identical tool visibility, permissions, and interfaces across models (standard tool suite + web_search/web_fetch + Clawhub skills).
- Models & serving
- 23 commercial API models accessed via OpenRouter; 12 open-weight models served locally with vLLM on a single NVIDIA RTX PRO 6000 (96 GB VRAM).
- Local models run in FP16 with 32K context; no quantization or distributed serving.
- Scoring & audit
- Tasks labeled auto/judge/hybrid with deterministic rules or an LLM judge following a rubric. Hybrid combines both with fixed weights.
- Audit uses deterministic triggers (e.g., empty artifacts, missing tool calls) to route suspicious positive judgements to a meta-judge that inspects execution traces and artifacts; unsupported positives are zeroed.
- Metrics
- Audited accuracy: Acc(m) = mean audited task scores across tasks applicable to model m (and Acc_g per group).
- Efficiency: average token usage; Cost(m) estimated from per-million-token API rates (αk) for input/output/cache tokens, averaged over tasks. Cost computed as sum_k αk * uk / 1e6.
- Pareto frontier analysis used to show accuracy vs. reference cost trade-offs by modality.
Implications for AI Economics
- Move beyond single aggregate scores when valuing models
- Procurement and deployment decisions should use workload-specific metrics: models differ by modality and execution form, so a single leaderboard ranking can mislead investment choices.
- Use workload-stratified accuracy and accuracy–cost frontiers to pick models that match the operational workload mix.
- Incorporate audited reliability into procurement and SLA design
- LLM-as-judge can award spurious credit (hallucinated correctness); evidence-based auditing materially changes measured performance. For contracts, evaluations that omit execution-trace auditing risk overestimating delivered capability and underestimating operational risk.
- Artifact contracts and time budgets (as used in DAREBench) are practical primitives to include in SLAs so buyers can verify delivered outputs, not just final text.
- Cost-performance trade-offs are modality-dependent
- Text vs. multimodal workloads have distinct Pareto frontiers; a model that is cost-effective on text may not be so for multimodal tasks. Economic evaluations should segment spending forecasts by modality and task type (e.g., single-step vs tool-mediated workflows).
- Local vs. API economics require careful TCO modeling
- Local open-weight models can match or beat APIs in some groups but generally trail frontier API models on accuracy-per-reference-cost. However, the benchmark treats local costs as third-party references — real TCO depends on hardware amortization, ops/engineering, energy, latency, scaling needs, and privacy/regulatory value.
- Decision-makers should model CAPEX (hardware + setup) vs OPEX (API tokens, maintenance), and account for differences in throughput, latency, and scalability for their workload profile.
- Tool-mediated workflows induce extra operational costs and risks
- Multi-step+tools tasks introduce runtime dependencies (external tools, web access) that affect both cost (extra tokens, tool invocation overhead) and failure modes. Economic forecasts should include tool integration/testing costs, monitoring, and remediation budgets.
- Practical recommendations for procurement and evaluation teams
- Evaluate models using representative workload matrices (like DAREBench’s 2×3) rather than single-task suites.
- Include evidence-auditing in vendor evaluations to avoid overpaying for illusory capability.
- Compute accuracy–cost Pareto frontiers by workload group to identify dominant choices and trade-offs.
- When considering on-prem vs cloud, run matched workload experiments and model full TCO (hardware lifecycle, staffing, availability requirements).
- Policy and market effects
- Benchmarks that emphasize verifiable artifact production and audited scoring could shift vendor claims toward more demonstrable SLAs, affecting pricing negotiation and competition.
- As open-weight models close performance gaps in some groups, competitive dynamics may favor hybrid strategies (API for frontier accuracy-critical workloads; local models for privacy-sensitive or latency-critical tasks).
Limitations to bear in mind for economic decisions - Cost estimates for local models are not measured operational costs but third-party references; real TCO can diverge substantially. - The single-GPU local configuration is a conservative on-premise setup and may not represent scaled deployment configurations. - The 233-task suite is curated for broad coverage but still a sample; organizations should run workload-specific pilots to validate vendor choices for their exact task mix.
If useful, I can: - Sketch a simple template to compute workload-specific TCO using DAREBench-style outputs (accuracy, tokens) and your cost assumptions. - Produce a one-page decision checklist for procurement teams that maps workload mix to recommended evaluation criteria and cost items to include.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports a large, systematic empirical evaluation (233 tasks, 35 models, 7,587 runs) under a shared runtime and includes an evidence-based audit; however, results are observational (benchmarking) and time-bound, rely on LLM judges for some scoring, and use reference cost estimates rather than measured economic outcomes, limiting claims about broader causal effects or real-world costs.
Methods Rigormedium — The authors implement a careful, reproducible evaluation: unified OpenClaw runtime, curated task selection, clear workload taxonomy, deterministic artifact contracts, three scoring modes, and a meta-judge audit to remove unsupported positives. Remaining concerns include potential selection bias in task curation, subjectivity and failure modes of LLM-as-judge, single‑GPU/local setup not representative of production on-prem deployments, use of reference (not measured) costs, and limited detail on inter-judge reliability and sensitivity analyses.
Sample233 tasks adapted from 22 existing benchmarks organized into a 2×3 workload matrix (text vs multimodal; single-step, multi-step, multi-step+tools); evaluated 35 models (23 commercial API models accessed via OpenRouter and 12 open-weight models run locally with vLLM on a single NVIDIA RTX PRO 6000 GPU); 7,587 applicable model–task runs; shared OpenClaw execution environment; scoring modes: automated, LLM-judge, hybrid; evidence-based meta-judge audit.
Themesadoption productivity
GeneralizabilityTime-bound snapshot: model landscape rapidly changes, so rankings and Pareto frontiers may not hold over time., Task selection bias: 233 curated tasks were filtered for measurability and difficulty, which may under- or over-represent certain real-world workloads., LLM-as-judge limitations: scoring relies on judge models and heuristics that may introduce systematic biases or miss some failure modes., Local deployment settings not representative: open-weight models are evaluated on a single-GPU FP16 32k-context configuration that differs from typical multi-GPU or optimized production deployments., Cost estimates are reference-based (per-token pricing) rather than measured end-to-end deployment costs (infrastructure, latency, engineering), limiting economic external validity., Toolset and execution environment constrained to OpenClaw and selected skills; other tool integrations or custom runtimes could change relative performance.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| DAREBench contains 233 tasks adapted from 22 source benchmarks and organizes them into a 2 × 3 workload matrix based on input modality and execution form. Other | positive | Benchmark coverage and workload stratification |
Reading fidelity
high
Study strength
high
|
n=233
2 × 3 workload matrix; 22 source benchmarks
|
| The study evaluates 35 models—23 commercial API models and 12 locally deployed open-weight models—over 7,587 applicable model–task runs. Other | positive | Scope of model evaluation |
Reading fidelity
high
Study strength
high
|
n=7587
35 models; 23 API and 12 local
|
| No single evaluated model dominates all workload groups. Output Quality | mixed | Accuracy across workload groups |
Reading fidelity
high
Study strength
medium
|
n=7587
|
| Text and multimodal workloads exhibit distinct accuracy–cost trade-offs and have different Pareto frontiers. Other | mixed | Model accuracy and estimated cost per task |
Reading fidelity
high
Study strength
medium
|
n=7587
|
| Locally deployed open-weight models are competitive with commercial API models in several workload groups but trail frontier commercial models overall. Output Quality | mixed | Accuracy across workload groups and deployment modes |
Reading fidelity
high
Study strength
medium
|
n=7587
|
| The evidence-based audit identified and removed 143 positive scores that were unsupported by execution evidence. Ai Safety And Ethics | negative | Unsupported positive scoring judgments |
Reading fidelity
high
Study strength
medium
|
n=7587
143 positive scores removed
|
| Under the reported task-group accuracy results, GPT-5.5 has the highest text average among the listed API models, while Claude-Opus-4.8 has the highest multimodal average. Output Quality | positive | Task accuracy averaged across text and multimodal workload groups |
Reading fidelity
high
Study strength
medium
|
n=7587
76.8% text average; 71.1% multimodal average
|
| DAREBench recommends selecting models based on workload profiles, deployment mode, and accuracy–cost trade-offs rather than relying on a single aggregate score. Decision Quality | positive | Model-selection and deployment decision quality |
Reading fidelity
high
Study strength
medium
|
n=7587
|