Multimodal judges commonly over-credit agent runs and break on hard cross-platform tasks; a new open benchmark and two OS-Shepherd reward models deliver near-frontier judging accuracy at dramatically lower cost, enabling large-scale, affordable CUA evaluation and training.

Main Finding

OSReward introduces a standardized, cross-platform benchmark for judging computer-using agent (CUA) trajectories and shows that (1) current vision‑language model (VLM) judges share a systematic leniency bias (accepting false successes), (2) the only judges reliable enough are too expensive to use at training/evaluation scale, and (3) open, low‑cost reward models (OS‑Shepherd 9B and 35B) trained on a newly released OS‑Shepherd‑100K corpus can match commercial judges’ accuracy on hard cases at ~30–60× lower cost.

Key Points

• Problem: Judging whether a long, interleaved trajectory of states/actions/reasoning actually fulfilled a task is critical for evaluation, curation, and RL, but human verifiers/annotators do not scale; VLM judges are used widely without systematic reliability checks.
• OSReward benchmark:
  • Built end-to-end on dedicated infrastructure across web, Windows, Ubuntu, and mobile with realistic, initialized environments and human‑vetted instructions.
  • 1,019 human-gold trajectories (long-horizon, up to ~100 steps).
  • Three views: full set (breadth), OSReward-Hard (challenge set of annotator-disagreed/hard cases), OSReward-Multi (binary verdicts plus fine-grained efficiency & alignment labels).
• Large evaluation: 27 VLM judges (including GPT/Gemini/Claude/Qwen variants) were benchmarked.
  • Frontier models perform well on the full set but degrade sharply on OSReward-Hard (best model < 70% accuracy; mean ~52%).
  • Consistent leniency error: judges more often accept runs where the agent claims success but the environment does not actually satisfy the task (false success).
• Cost–accuracy tradeoff:
  • Reliable judges (e.g., Claude-Opus-4-8, GPT-5.5) are expensive to run at scale; cheaper open models perform worse.
  • OS‑Shepherd reward models (9B, 35B), trained on OS‑Shepherd‑100K, sit near the cost–accuracy Pareto frontier — matching commercial judge accuracy on hard cases at a fraction (~30–60× lower) of the cost per 1,000 trajectories.
• Data release: OS‑Shepherd‑100K — ~100K reasoning-annotated trajectory judgments curated from ~300K judge instances — with failure-type annotations to support open research.

Data & Methods

• Infrastructure and collection:
  • Cross‑platform testbeds with realistic initial state: fully provisioned desktop/mobile/Ubuntu machines, seeded files, logged-in apps, and live websites.
  • Annotators authored and peer‑screened ~1,500 instructions; ≈800 used for rollouts.
  • Agents from multiple backbone families (Claude, Gemini, Kimi, Qwen) executed instructions to generate diverse real successes and failures.
  • Pre‑filtering removed execution problems (blocks, crashes) before human labeling.
• Human annotation pipeline:
  • Each trajectory reviewed in full (screenshots, actions, chain-of-thought).
  • Strict labeling rule: outcome counts as success only if obtained/verified via the environment.
  • Three independent annotators per trajectory; disagreements escalated to meta-review. Successes get efficiency and alignment scores (3-class, 0/0.5/1); failures receive taxonomy labels (action, perception, planning/reasoning, memory).
  • OSReward-Hard constructed from annotator-split cases to concentrate genuinely difficult instances.
• OS‑Shepherd data & models:
  • OS‑Shepherd‑100K assembled from a large automated collection (300K judge instances filtered into ~100K), annotated without new human labeling but guided by judge-study findings (to address prominent failure modes).
  • Two-stage model training: (1) build accurate judging, (2) explicitly target the “false success” bias.
  • Models released: OS‑Shepherd‑9B and OS‑Shepherd‑35B (self-hostable, open).
• Evaluation metrics: binary accuracy on verdicts; recall for GT=FAIL vs GT=SUCCESS to characterize strictness/leniency; cost per 1k trajectories vs accuracy to analyze the Pareto frontier.

Implications for AI Economics

• Lower marginal cost for large-scale evaluation and RL:
  • Reliable open reward models at 30–60× lower run cost materially reduce the computational/economic barrier to performing millions of trajectory judgments. This lowers the cost of large-scale RL training loops, data curation, continuous evaluation, and automated filtering pipelines.
  • Research groups and smaller firms can feasibly run more iterations and larger-scale experiments, accelerating innovation and lowering incumbent advantage rooted in compute budget.
• Market and competition dynamics:
  • Demonstrates a clear commercial opportunity and competitive pressure: high‑accuracy commercial judge APIs command premium prices, but open, cheaper alternatives that reach comparable accuracy on hard cases can shift demand toward self-hosted or open solutions.
  • Vendors may respond by differentiating via improved calibration, debiasing, or integration of specialized verifiers; alternatively, price competition could intensify in the judge-as-a-service niche.
• Data- and quality‑externalities:
  • The prevalent leniency bias across many judges implies systematic label noise in datasets curated using off-the-shelf judges. If uncorrected, this can lead to trained agents that overfit to claimed successes (policy degradation), increasing downstream costs (more RL iterations, manual fixes).
  • Cheap judges reduce annotation costs but risk amplifying biases if those biases are not corrected (OS‑Shepherd explicitly targets false-success bias — an example of necessary defensive investment).
• Incentives for standardization and benchmarking:
  • OSReward provides a standardized way to evaluate judge reliability that can be incorporated into procurement and model‑auditing decisions. Buyer organizations (platforms, model consumers) can use such benchmarks to choose judges that balance cost and risk.
  • Standard evaluation reduces transaction costs in the market for reward models: purchasers can compare cost/accuracy tradeoffs meaningfully.
• Labor and organizational impact:
  • Scaled use of reliable automatic judges can reduce reliance on human annotators, shifting labor from routine labeling to higher‑value time (annotation auditing, edge-case review, dataset design). This changes where human effort is allocated and may reduce annotation budgets long-term.
• Policy and risk management:
  • Because judge bias systematically favors false successes, organizations should budget for guardrail costs (spot human audits, stricter verifiers for critical tasks) even when using low-cost judges. For high-stakes applications (security screens, finance, compliance), reliance on cheap judges without additional checks risks costly failures.
• Research and investment signals:
  • The demonstrated returns to targeted dataset/model design (OS‑Shepherd‑100K + focused training on false successes) indicate high economic value from relatively small, well-designed labeling efforts plus model specialization. Investors and labs may prefer funding focused reward-model datasets and debiasing pipelines over simply scaling model size.
• Cautions for adopters:
  • The benchmark and models are strong steps forward, but generalization beyond the covered platforms/task distributions requires scrutiny. Overreliance on a single judge or on automated labels without complementary verification may externalize costs later in the product lifecycle.

Summary takeaway: OSReward quantifies a previously underappreciated reliability vs cost tradeoff in automated trajectory judging, exposes a systematic leniency bias that can degrade downstream training, and supplies an open, cost‑effective mitigation (OS‑Shepherd models + OS‑Shepherd‑100K) that materially lowers the economic barrier to large‑scale, reliable reward signals—shifting the economics of agent evaluation and RL pipelines in favor of broader, cheaper experimentation while highlighting the need to manage bias‑related risks.