Treating commercial CAD as COM-call programs, not pixel-control chores, dramatically improves autonomous task success: GUI-driven agents fail on industrial CAD, while COM execution plus a self-correcting agent delivers substantial, robust gains on a new real-world benchmark.

Main Finding

ComAct (COM-as-Action) reframes professional-software manipulation as deterministic program synthesis over the Windows Component Object Model (COM). Using COM as a unified action space and executing generated Python/COM scripts in real Windows CAD environments leads to large, immediate gains in reliability and task success compared with GUI-based and ad-hoc API approaches. The authors validate this in the demanding domain of industrial CAD via (1) ComCADBench, a 1,000-task benchmark across SolidWorks/Inventor/AutoCAD; (2) ComForge, a scalable platform for thousands of Dockerized Windows environments; and (3) ComActor, a self-correcting agent trained with a three-stage pipeline (SFT → agentic SFT → RL with geometric reward). ComActor achieves state-of-the-art results on ComCADBench (high Valid Rate and Success Rate across single- and multi-task settings) and generalizes to external CAD benchmarks.

Key Points

• Motivation
  • GUI-based agents (screenshot + mouse/keyboard) are fragile in dense professional UIs and suffer cascading long-horizon error accumulation.
  • Existing API/MCP approaches are fragmented, application-specific, and often unavailable for closed-source commercial software.
  • MLLMs are strong code generators; COM provides a widely supported, semantic programmatic interface across Windows professional applications.
• The ComAct paradigm
  • Treat actions as executable COM scripts (Python via win32com) rather than atomic GUI operations.
  • Reserve visual perception for coarse observation/verification; use code execution for deterministic state transitions.
  • Enables cross-application programmatic workflows and avoids hundreds of brittle GUI steps.
• System components
  • ComCADBench: first benchmark for agents operating real CAD COM interfaces; 1,000 tasks spanning 7 engineering activities (3D modeling, 2D sketching, assembly, drawing, editing, mass properties, interference detection), split into Single-Task (400) and Multi-Task (600) long-horizon pipelines.
  • ComForge: Dockerized Windows containers hosting real CAD apps, exposing a small unified API (code_execute, gui_action, reset), with an asynchronous scheduler enabling 1000+ parallel environments for large-scale training and evaluation.
  • ComActor: agent trained via a progressive three-stage framework:
  • Stage 1 — supervised fine-tuning (SFT) on a verified instruction→COM-code corpus (syntactic correctness).
  • Stage 2 — agentic SFT using multi-turn interaction traces and multimodal feedback; failed rollouts corrected by a strong teacher to teach iterative debugging.
  • Stage 3 — group-normalized RL (GRPO) optimizing a continuous geometric reward based on Chamfer Distance between generated and ground-truth artifacts (task-level fidelity).
• Data construction
  • Synthesized verified instruction–code pairs by prompting MLLMs on public CAD datasets (SketchGraphs, Text2CAD, Fusion360Gallery), then executing generated scripts in isolated containers and retaining only validated examples.
• Empirical results
  • GUI-based agents: near-zero task success on ComCADBench.
  • COM-based few-shot prompting: large immediate improvements (varies by model), but still limited and unstable.
  • ComActor (full pipeline): state-of-the-art performance — high Code Valid Rates and Task Success Rates across single- and multi-task categories (e.g., overall high 80–90%+ range on many tasks in Table 1), strong resilience on long-horizon pipelines where GUI and baseline methods collapse.
  • Generalization: ComActor transfers to external CAD datasets (Text2CAD, CADPrompt).
• Limitations called out
  • COM is Windows-centric; reliance on vendor exposure of COM interfaces.
  • Syntactic correctness of code does not guarantee geometric fidelity → necessitates task-level RL and verification.
  • Requires substantial compute/infrastructure (ComForge) for large-scale training and validation.

Data & Methods

• Benchmarks and evaluation
  • ComCADBench: 1,000 verified tasks across SolidWorks, Inventor, AutoCAD; each task includes a natural-language instruction and ground-truth artifact.
  • Metrics:
    • Valid Rate (VR): % of executed COM scripts that run error-free.
    • Success Rate (SR): % of final artifacts meeting task-specific criteria.
  • Task verification:
    • 3D modeling & assembly: Chamfer Distance (CD) threshold (e.g., CD ≤ 1e-3).
    • 2D sketch evaluation via DXF-based CD with strict thresholds.
    • Engineering drawings require presence of required views/annotations.
    • Mass properties & interference detection compared element-wise to ground truth JSON.
• Training data construction
  • Start from structured geometry specs in public datasets → prompt an MLLM to produce an instruction and a corresponding COM script → execute script in containerized CAD environment → keep only executions that reproduce the target geometry → yields a verified SFT corpus.
• Model & training pipeline
  • Stage 1 (SFT): train model to map instruction → COM script (autoregressive objective). Goal: syntactic & API-correct code generation.
  • Stage 2 (Agentic SFT): collect multi-turn interaction trajectories by rolling out Stage-1 policy. Successful rollouts are used directly; failed rollouts are corrected by a teacher model which produces reasoning traces and revised code. Train agent to generate reasoning tokens, decision tokens (CODE/DONE/FAIL), and code conditioned on multimodal history (screenshots + terminal feedback).
  • Stage 3 (GRPO): on-policy RL optimizing continuous geometric reward Rgeo based on Chamfer Distance (log-linear transformation to avoid sparsity). Use group-normalized advantages and clipped policy updates with a KL penalty toward a reference policy.
• Infrastructure (ComForge)
  • Each environment: Docker container encapsulating a Windows VM with CAD software pre-installed and a daemon to run code_execute/gui_action/reset.
  • Observations: screenshot + terminal output after each action.
  • Asynchronous scheduler aggregates and routes requests across thousands of containers for scalable data collection and RL.
• Reward design
  • Continuous geometric reward derived from Chamfer Distance between generated artifact and ground truth meshes/curves, with thresholds for near-perfect and failure cases; execution failures yield zero reward.

Implications for AI Economics

• Productivity and cost structure
  • Large potential productivity gains in high-skill professional software workflows (CAD, multi-suite engineering tasks) by converting many manual GUI steps into single deterministic scripts; reduces time-per-task and error rates.
  • Lower marginal cost for executing complex design pipelines (once models and infrastructure are in place) — shifting fixed investment (model training and ComForge-like infrastructure) toward lower variable labor costs.
• Labor market effects
  • Substitution/complementarity: routine and repetitive CAD operations (parametric modeling, assembly checks, mass-property computations) are highly automatable; roles focused on those tasks may shrink or shift toward oversight, validation, and higher-level design reasoning.
  • Upskilling demand: human workers will need skills in prompt/spec formulation, verification, and supervising agents; higher value placed on system integration and cross-domain design planning.
• Market structure and value capture
  • Firms that invest early in agent infrastructure (models + execution platforms) can capture outsized returns via reduced design cycle time and higher throughput — potential winner-take-most dynamics in design-heavy industries.
  • Software vendors (e.g., CAD/Office/Adobe) exposing rich COM interfaces increase third-party automation opportunities, but they also gain bargaining power: they could monetize access (licensing, API restrictions) or limit integration, affecting who captures economic gains.
• New business models
  • Agent-as-a-service (AaaS) for professional workflows: companies could offer turnkey automation for CAD pipelines (design generation, compliance checks, mass property reporting) as a subscription or per-run service.
  • Market for verified instruction→script corpora and fine-tuned agents tailored to verticals (automotive, aerospace, architecture).
• Investment & barrier-to-entry considerations
  • High fixed costs: building ComForge-scale parallel execution environments and training data (verified executions) requires capital and engineering resources; this may favor incumbents or well-funded startups.
  • However, once a validated agent exists, scaling deployments is cheaper — so economies of scale matter.
• Competitive and regulatory risks
  • Vendor control of programmatic hooks: software providers could restrict COM or charge for privileged API access, creating platform risk and rent extraction.
  • Liability & IP: automatically generated designs may implicate intellectual property, standards compliance, or safety regulations (e.g., structural/mechanical failures). Firms will need governance frameworks and human-in-the-loop validation to manage liability.
  • Labor regulation and social policy: automation of skilled tasks raises questions about retraining subsidies and transition assistance for affected workers.
• Externalities and diffusion
  • Faster iteration cycles could accelerate innovation in product development, but also compress lead times and intensify competition.
  • Lower entry costs for complex engineering (if ComAct-like agents are widely accessible) may democratize certain design activities and create new small-actor competition.
• Measurement and evaluation for policy/economics
  • Use benchmarks such as ComCADBench to quantify productivity gains and to compare agents’ economic value (time saved per task, error reduction, headcount displacement).
  • Economic impact studies should consider both direct savings and second-order effects (market concentration, changes in skills demand, vendor pricing power).
• Recommendations for stakeholders
  • Firms: invest selectively in building validated execution pipelines and verification layers; prioritize tasks with high repetition and well-specified verification metrics.
  • Software vendors: design API access and licensing models that balance monetization and ecosystem growth; consider certified integration programs.
  • Policymakers: monitor labor displacement risks in skilled occupations and support retraining; consider standards for agent-generated artifacts where safety is critical.
  • Researchers: evaluate cross-domain generality, cost-benefit analyses, and the effects of platform strategies (open vs closed COM access) on diffusion.

Overall, ComAct demonstrates a pragmatic technical pathway to automate high-value professional software tasks by treating COM as a deterministic action space. Economically, this can materially shift productivity, firm strategy, and labor dynamics in design- and software-intensive industries — but the distributional outcomes will depend on software-platform policies, investment barriers, and governance choices.