A new benchmark for professional software exposes sharp limits of current multimodal agents: on 436 realistic tasks across 13 applications the best agent succeeds on only 24% of intermediate tasks and fails complex multi-app workflows, underscoring major barriers to deploying agents in real professional work.

Main Finding

ProSoftArena introduces a realistic, VM-hosted benchmark and evaluation platform for multimodal agents in professional software. It shows that current multimodal agents can handle basic GUI operations (L1) but perform poorly on software-level tasks (best agent: 24.4% success on L2) and essentially fail on multi-application workflows (L3). Key bottlenecks are long-horizon planning, domain knowledge, state tracking, and robust visual grounding; human-in-the-loop collaboration helps but agents rarely ask for help.

Key Points

• Capability hierarchy (L1–L5):
  • L1 Operation: atomic GUI manipulations.
  • L2 Software: multi-step software-specific tasks requiring domain knowledge.
  • L3 Pipeline: cross-application workflows and data transfer.
  • L4 Creative: open-ended creative generation using L2/L3 skills.
  • L5 Project: long-horizon, multi-stakeholder industrial/scientific projects (future work).
• Benchmark scope:
  • 436 domain-authentic tasks across 6 disciplines, 20 subfields, and 13 professional applications (e.g., Photoshop, SolidWorks, ArcGIS, Excel, ChemDraw).
  • Tasks crafted by domain experts and mapped to capability levels; human performance recorded (steps/time).
• Platform and evaluation:
  • Executable real-computer environment: Windows 11 VMs managed by Docker; fixed software versions; deterministic snapshots for reproducibility.
  • Agents interact via screenshots + system signals and perform keyboard/mouse actions.
  • Execution-based automated evaluation: per-task scripts verify final system state / output artifacts (e.g., file checks, MSE on images).
  • Human-in-the-loop modes: (1) Human-initiated takeover; (2) Agent-initiated assistance.
• Experimental results and error analysis:
  • Good L1 performance, sharp drop at L2 (best ~24.4%); L3 multi-software workflows fail in experiments.
  • Common failure modes: planning errors, insufficient domain knowledge, visual grounding mistakes, poor state tracking.
  • Ablations: richer visual inputs, domain priors, and longer action history improve performance but increase compute cost.
  • Agents rarely request help; human-agent collaboration demonstrably beneficial when used.
• Benchmark availability: project site and dataset released for community use.

Data & Methods

• Task curation: 436 tasks designed by domain experts to reflect authentic professional workflows; each task includes instruction, initial VM state, and success criteria.
• Environment:
  • Windows 11 VMs with 13 core professional applications pre-installed to fixed versions.
  • Initialization scripts restore snapshots, load files, and set window/tool states to ensure consistent starting contexts.
• Agent loop and formalization:
  • Interaction modeled as a POMDP with MLLMs as policy models producing GUI actions (mouse/keyboard) and natural-language planning.
  • Per-task max-step limits and DONE/FAIL termination actions.
• Evaluation:
  • Execution-based assessment: specialized evaluation functions per task inspect internal state / artifacts (e.g., configuration files, exported results, pixel-difference thresholds).
  • Human-in-the-loop experiments run both takeover and assistance modes; human monitoring recorded.
• Baselines and experiments:
  • Evaluated multiple state-of-the-art proprietary and open-source multimodal agents and specialized computer-use agents.
  • Quantitative metrics: task-level binary success (1.0/0.0), aggregated by capability level; human performance benchmarks for reference.
  • Ablation studies on visual input richness, domain priors, action history length, and help-seeking behavior.

Implications for AI Economics

• Near-term automation potential is limited and granular:
  • L1 tasks (atomic GUI ops) are the most immediately automatable — firms can realize incremental productivity gains by automating repetitive UI steps.
  • L2–L3 tasks (software workflows and cross-application pipelines) remain challenging; broad automation of professional work is not imminent given current ~24% L2 and ~0% L3 success.
• Complementarity and labor impacts:
  • Agents are likely to act as augmentative tools rather than direct substitutes in the near–medium term. Human experts will remain essential for planning, domain judgment, QA, and cross-application coordination.
  • Demand will increase for workers who can supervise, correct, and orchestrate agent outputs (human-in-the-loop skills), shifting labor toward oversight, integration, and higher-order decision tasks.
  • Routine, well-specified L1 tasks may experience downward pressure on wages or be reallocated; premium for domain-specialist and project-coordination skills may rise.
• Adoption and firm investment considerations:
  • Firms should prioritize deploying agents in tightly-scoped, low-risk tasks with clear verification and rollback (e.g., via VM snapshots), investing in domain priors, curated training on proprietary workflows, and UX for human-agent handoffs.
  • Upfront costs can be substantial: licensing professional software, VM infrastructure, environment setup, and building task-specific evaluation scripts. ROI depends on error rates and the cost of mistakes in domain workflows.
• Market structure and platform effects:
  • Proprietary agents (integrated with commercial stacks) vs. open-source alternatives → potential for vendor lock-in around integrated toolchains and specialized agent capabilities; firms may prefer agents validated on execution-based benchmarks.
  • Standardized, reproducible benchmarks like ProSoftArena reduce asymmetric information about agent capabilities and can accelerate trustworthy adoption.
• Productivity measurement and standards:
  • Execution-based evaluation aligns agent claims with real-world deliverables; firms and regulators may require similar verification before deploying agents in high-stakes professional settings.
  • Productivity gains should be measured net of human oversight and error-correction costs — naive task-success rates overstate usable automation value.
• Long-run dynamics:
  • As agents improve on L2/L3 (via better long-horizon planning, domain representations, and API integrations), the marginal cost of routine professional tasks will fall, enabling smaller firms to access skills previously concentrated in specialists and reshaping competition.
  • Investments that reduce cross-software friction (APIs, standardized file formats, improved accessibility hooks) will accelerate automation and substitute work that is currently coordination-heavy.
• Policy and liability:
  • High-cost errors in professional domains (engineering, medicine, legal) warrant regulation, certification, and mandatory human oversight; this raises compliance costs and slows full automation adoption.
• Research and market signals:
  • Benchmark shows where R&D funding should focus for highest economic returns: persistent weaknesses (planning, state tracking, domain knowledge, calibrated uncertainty/ask-for-help behavior) are the technical bottlenecks whose removal would unlock substantial economic value.
  • Firms and investors should assess agent readiness using execution-based, domain-specific benchmarks rather than proxy tests on generic desktop tasks.

Overall, ProSoftArena provides an empirically grounded map of where multimodal agents currently add value and where they fall short in professional software workflows — information crucial for firms deciding where to adopt, invest, or regulate AI agents in high-value, domain-specific tasks.