UI-Mate, an open-weight GUI agent trained with executable environments and demonstration guidance, sets a new open-weight state of the art on desktop task benchmarks and markedly improves long-horizon reliability—one same-task demonstration lifts strict success on a 33-task subset from 17.2% to 35.4% and progress from 67.9% to 81.1%.

Main Finding

UI-Mate is an open-weight foundation GUI agent that combines an environment-grounded, closed-loop training pipeline with an in‑context multimodal demonstration method (DemoCUA). This combination materially improves long‑horizon, cross‑application task reliability: UI‑Mate‑27B sets new open-weight state-of-the-art results on multiple computer-use benchmarks and, critically, demonstrates that single demonstrations substantially increase strict success and progress on long, underspecified office workflows.

Key quantitative results (selected) - UI‑Mate‑27B: 77.0% OSWorld‑Verified, 66.2% WindowsAgentArena. - OSWorkerBench (100 long‑horizon office tasks): 41.0% strict success, 76.9% progress. - Gains vs Qwen3.6‑27B base: +17.7 percentage points (strict success) and +24.5 pp (progress) on OSWorkerBench. - Demo effect (33-task self-demo subset): strict success 17.2% → 35.4%; progress 67.9% → 81.1% with one same-task demonstration. Project page: https://ui-mate.github.io

Key Points

• Two central bottlenecks addressed:
  • Training-level: scarcity and distributional skew of executable GUI trajectories (short single-app tasks dominate; long-horizon, multi-app, and recovery behaviors are underrepresented).
  • Interaction-level: prompt ambiguity and underspecified procedural choices in real user workflows → inconsistent execution.
• UI‑Mate combines:
  • Environment-grounded data and verifiers to ensure trajectories are executable and check outcomes.
  • Supervised fine-tuning (SFT) plus online agentic reinforcement learning (RL) on verifiable tasks.
  • DemoCUA: converts multimodal demonstrations into subtask-level workflows that serve as flexible guidance (not rigid replay); live screenshots remain authoritative.
• OSWorkerBench: new benchmark of 100 long‑horizon office workflows (41 normalized apps). It supports:
  • Instruction-only evaluation.
  • Demonstration-guided evaluation with two demo types: self-demos (same-task strong-agent rollouts; 33 targets) and variant-demos (human recordings of related tasks; 45 targets).
• Demonstrations improve both mean completion and consistency: they help the agent resolve underspecified procedural choices more reliably (less variance across runs).
• UI‑Mate includes a deployment harness (UI‑Mate App) to record demos and execute agents on real desktops.

Data & Methods

Data pipeline and environment grounding - Closed-loop pipeline: task generation → environment construction (apps, files, init states) → massively parallel rollouts → trajectory filtering → capability diagnostics → targeted rebalancing. - Trajectories are paired with executable verifiers; pipeline yields task–verifier bundles usable both for SFT and for RL. - Capability tree: hierarchical extraction of capabilities (apps, operations, workflows) used to diagnose corpus coverage and rebalance data collection toward underrepresented skills.

Training stack - Supervised fine-tuning: teaches interaction protocol, visual grounding, multi-step planning and cross-app information flow. - Agentic RL (online): uses verifiers to provide trajectory-level rewards; introduces mechanisms tailored to GUI RL: - Trajectory-to-token credit assignment to align sequence modeling outputs with episodic RL rewards. - Asynchronous group-relative optimization for scaling training across many parallel environments. - Adaptive curriculum sampling to focus RL collection on weak capabilities identified by the capability tree.

DemoCUA (in-context demo learning) - Demonstration representation: a demonstration d is segmented into ordered subtasks s_n = (ℓ_n, v_n, u_n), where ℓ_n is subtask goal (text), v_n is a verifiable completion criterion, and u_n is an ordered set of textual, visually-grounded action descriptions (no pixel coordinates). - During execution the model is given gt = Φ(d, n_t): progress checklist (all subtask goals labeled completed/current/upcoming), plus the detailed action descriptions only for the active subtask. - The live screenshot is authoritative: the agent follows demonstrated steps where they match the live state, supplies omitted low-level actions, and re-plans when the demonstration diverges. - Training covers cases where demos agree, diverge, or are irrelevant to the live state, teaching the model to treat demos as priors, not scripts.

Benchmark & evaluation - OSWorkerBench: 100 tasks, including Long‑Memory (delayed reuse) and Multi‑App (information transfer across ≥3 apps). - Evaluation protocols include strict success (executable verifier) and progress metrics. - Paired design for demo experiments: same verifier, initialization, and instruction; only demonstration availability/type changes to isolate demo value.

Implementation & deployment - UI‑Mate‑27B is the reported open-weight model; system supports recording demos (human or agent) and running on users’ desktops with a harness that supplies workflow state and live screenshots.

Implications for AI Economics

Productivity and labor market effects - Increased reliability on long, cross‑application tasks makes GUI agents more deployable for knowledge‑work automation (e.g., recruiting workflows, reporting, document handling). - The demonstration mechanism reduces per-user personalization costs: a single multimodal demo can substitute for expensive fine-tuning or repeated prompt engineering, lowering marginal costs of customizing agents for individual workflows. - Likely short-to-medium term effect: augmentation of administrative, clerical, and coordination tasks—raising productivity and possibly compressing demand for routine office labor while increasing demand for oversight, prompt/design, and integration roles.

Cost structure and scaling - Environment‑grounded pipelines and massive parallel RL require substantial engineering and compute, but they reduce human labeling by enabling automated rollout + verification. Economically, trade-offs: - High fixed costs (engineering, compute clusters) versus lower marginal costs per additional task/environment once pipeline is in place. - Open-weight availability (UI‑Mate is open-weight) lowers barriers to adoption and stimulates competitive downstream services, potentially accelerating diffusion.

Value of in-context demos vs fine-tuning - Demonstrations provide a low-friction route to customization (user records a demo), which can be cheaper and faster than client-specific fine-tuning or building specialized rule systems. - For firms, this implies faster deployment and lower integration costs; for gig/consulting markets, demand may shift toward creating high-quality demonstration libraries and verification infrastructures.

Market and competition dynamics - Open-weight SOTA reduces lock-in to closed-source providers and can spur a broader ecosystem of marketplace tools (demo sharing, verifier development, verticalized benchmarks). - Firms with capability to operate environment-grounded pipelines and RL at scale maintain competitive moats (engineering + compute + curated environment libraries).

Policy, privacy, and externalities - Live demos and desktop access create sensitive data flows (files, emails, applicant data in the example) → privacy, compliance, and security risks with economic consequences (liability, compliance costs). - Need for regulatory and contractual safeguards, auditability of verifiers, logging, and human-in-the-loop approval for high‑stakes actions. - Reliability gains reduce the necessity for continuous human monitoring but increase the consequences of systematic failures; insurance, certification, and standards may emerge.

Research and measurement implications for economists - Natural experiments: staged rollouts (with/without demo capability) allow measuring causal effects of deployment on productivity and labor demand in firms. - Cost‑benefit analyses should include engineering fixed costs, benchmarking reliability (not just mean success), and externalities (privacy, error risk). - Distributional effects: democratization via open-weight models may spread productivity gains unevenly—follow-on research should measure access gaps across firm sizes and sectors.

Caveats / limitations to keep in mind - Benchmarks (even OSWorkerBench) are controlled and may understate real-world heterogeneity of user file systems, personalized configurations, and adversarial edge cases. - Demonstrations evaluated include same-task self-demos (strong gains) and variant-demos (transfer harder); real users will often provide imperfect demos—robustness to diverse mismatch levels matters for economic impact. - Engineering complexity and compute intensity mean practical adoption depends on organizational resources.

Summary recommendation for stakeholders - For firms: consider pilot deployments of demo-guided GUI agents for routine, high-volume workflows where verification is feasible and privacy controls are in place; expect faster ROI when leveraging reusable demonstration libraries. - For policymakers and labor analysts: monitor occupational impacts in administrative and clerical roles and invest in retraining for higher-value supervision and integration tasks. - For researchers: evaluate demo generalization across heterogeneous real-world environments, measure downstream economic impacts, and develop standards for verifiers and privacy-preserving demo collection.