4 cumulative citations
View corpus contextEnterprise workflows stump current multimodal LLM agents: on a new 1,756-task benchmark across CRM, ITIL and ERP, top models succeed on under half of tasks, leaving a wide gap to human performance and signalling the need for domain-specific agent development.
EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents
EntWorld is a large-scale, schema-grounded benchmark of 1,756 enterprise tasks (CRM, ITIL, ERP, etc.) with SQL-based verification that reveals state-of-the-art multimodal LLM agents achieve only ~47.6% success, indicating a sizable capability gap versus humans for enterprise workflows.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Recent advances in Multimodal Large Language Models (MLLMs) have enabled agents to operate in open-ended web and operating system environments. However, existing benchmarks predominantly target consumer-oriented scenarios (e.g., e-commerce and travel booking), failing to capture the complexity and rigor of professional enterprise workflows. Enterprise systems pose distinct challenges, including high-density user interfaces, strict business logic constraints, and a strong reliance on precise, state-consistent information retrieval-settings in which current generalist agents often struggle. To address this gap, we introduce EntWorld, a large-scale benchmark consisting of 1,756 tasks across six representative enterprise domains, including customer relationship management (CRM), information technology infrastructure library (ITIL), and enterprise resource planning (ERP) systems. Unlike previous datasets that depend on fragile execution traces or extensive manual annotation, EntWorld adopts a schema-grounded task generation framework that directly reverse-engineers business logic from underlying database schemas, enabling the synthesis of realistic, long-horizon workflows. Moreover, we propose a SQL-based deterministic verification mechanism in building datasets that replaces ambiguous visual matching with rigorous state-transition validation. Experimental results demonstrate that state-of-the-art models (e.g., GPT-4.1) achieve 47.61% success rate on EntWorld, substantially lower than the human performance, highlighting a pronounced enterprise gap in current agentic capabilities and the necessity of developing domain-specific agents. We release EntWorld as a rigorous testbed to facilitate the development and evaluation of the next generation of enterprise-ready digital agents.
Summary
Main Finding
EntWorld is a new, large-scale, privacy-free enterprise GUI environment and benchmark (1,756 tasks across six enterprise apps) that (1) programmatically synthesizes realistic, long-horizon, stateful workflows by reverse‑engineering database schemas, and (2) deterministically verifies agent success via SQL state checks. Evaluation shows a large “enterprise gap”: state-of-the-art multimodal agents (e.g., GPT-4.1) perform substantially below human experts (GPT-4.1 SR 47.61%; best agent EntAgent-RL SR 56.89%; human ≈85%), indicating substantial room and economic incentive for domain-specific agent development.
Key Points
-
Scope and domains
- 1,756 instantiated intents spanning 6 open-source enterprise apps: EspoCRM (320), ZenTao (362), OpenProject (278), Veops CMDB (302), iTOP (397), Snipe‑IT (97).
- Tasks emphasize enterprise characteristics: high-density UIs, long-horizon state dependencies, strict business logic constraints.
-
Data generation & verification innovations
- Schema-driven task synthesis: pipeline reverse-engineers DB schemas to infer table semantics, relations, and abstract workflows; LLMs are used for schema JSON generation and template synthesis.
- Deterministic SQL-based verification: tasks come with ground-truth SQL and are checked by executing verification SQL in transactions (with rollback), replacing brittle visual or LLM-based judging.
-
Environment & interface
- Dockerized, multi-app sandbox providing screenshots + accessibility tree (DOM), action space simulating keyboard/mouse (click/type/scroll/etc.), and WebArena-compatible task format.
- Tasks tagged to support environment reset and deterministic evaluation.
-
Benchmark and metrics
- Quantifiable difficulty metric (D_task) combining structural and operational factors (table/relational/operation/SQL complexity).
- Evaluation uses task Success Rate (SR). Example results (on 1,756 tasks): GPT-4.1 47.61% SR; Claude 3.5 Sonnet ~28.8%; UI‑TARS 34.51%; EntAgent‑SFT 50.57%; EntAgent‑RL 56.89%; human ≈85%.
-
Agent training takeaway
- Domain-specific training (SFT + RL) substantially improves performance; EntAgent‑RL outperforms generalist proprietary models on this benchmark.
- Visual grounding and implicit business logic remain major failure modes for current agents.
Data & Methods
-
Dataset construction pipeline
- Infrastructure: Dockerized instances of enterprise apps, DatabaseAdapter abstraction to support multiple DB backends.
- Workflow discovery: schema parsing → LLM generates SchemaJSON per table → relation inference via explicit FK parsing + LLM-inferred implicit links → SQL probes to verify inferred relations.
- Template synthesis + instantiation: LLMs generate bilingual prompts, SQL logic, placeholders; real DB content used to instantiate task inputs; CUD tasks validated by executing verification SQL in transaction blocks and rolling back.
- Filtering & standardization: duplicate filtering, illegal-answer removal, WebArena-format conversion, require_reset tagging for stateful tasks.
-
Verification
- SQL execution-based checks (deterministic) validate that target state transitions occurred (row inserts/updates/deletes, query results), removing ambiguity inherent to pixel/LLM judging.
-
Experiments
- Baselines: proprietary (GPT‑4.1, Claude 3.5 Sonnet), open-weight multimodal agents (UI‑TARS, Qwen‑VL models, WebRL variants), and two in-house agents (EntAgent‑SFT and EntAgent‑RL).
- Training: EntAgent variants trained on trajectory data derived from templates; EntAgent‑RL used RL fine-tuning on successful trajectories.
- Metric: Success Rate (task completion as verified by SQL). Cost-per-task reported for some proprietary models (e.g., GPT‑4.1 $0.16/task).
Implications for AI Economics
-
Market and ROI signals
- Large enterprise gap implies commercial value for specialized agents: firms can justify investment in domain‑tuned models (SFT + RL) to achieve meaningful lift over generalist models.
- Cost trade-offs: the paper reports per-task inference costs for proprietary APIs (e.g., GPT‑4.1 $0.16/task). Open-weight agents with targeted fine-tuning may yield superior performance at potentially lower operational costs—important for large-volume automation use cases.
-
Labor substitution and augmentation
- Current agents reduce but do not eliminate human labor: best SR ~57% vs human ~85%. Near-term deployments are more likely to assist or augment knowledge workers rather than fully automate complex enterprise workflows.
- High-stakes tasks (billing, compliance) require deterministic verification and audit trails; EntWorld’s SQL verification model maps directly to governance/compliance needs and reduces liability, improving firms’ willingness to adopt automation.
-
Deployment, compliance, and vendor evaluation
- Deterministic benchmarks like EntWorld enable purchasers and auditors to quantify agent competence on enterprise-relevant tasks before procurement, supporting procurement decisions and contractual SLAs.
- Sandbox/dockerized, privacy-free benchmarks lower barriers for vendors and enterprises to test agents without using proprietary data—this can accelerate vendor evaluation cycles and encourage marketplace competition.
-
Investment priorities for maximizing economic impact
- Focus R&D on UI grounding (high-density interfaces), schema-aware reasoning, transaction-safe execution, and integration with backend state (DB/API verification).
- Invest in domain-specific RL fine-tuning pipelines and curated trajectory generation to close the gap faster than relying on generalist LLMs.
- Build deterministic verification and auditability into agent products to satisfy compliance-heavy industries (finance, healthcare, enterprise IT), which may command premium adoption rates.
-
Limitations & caution
- EntWorld is synthesized from open-source apps; real enterprise systems can be more heterogeneous and access-controlled—so measured SRs are indicative but may be optimistic/limited in representing all production contexts.
- Economic assessments should include costs of integration, fine-tuning, maintenance, and legal/compliance risk mitigation, not just per-task inference cost.
Overall, EntWorld provides a practical, auditable yardstick for trading off agent performance, deployment cost, and compliance risk—helpful for firms and investors assessing where to allocate resources in enterprise automation.
Assessment
Paper Typedescriptive
Evidence Strengthn/a — This is a benchmark/dataset paper, not an empirical causal study — it reports model evaluation results rather than making causal claims about economic effects.
Methods Rigormedium — The dataset construction uses a clear, reproducible schema-grounded task synthesis pipeline and deterministic SQL-based state-transition verification (strengths). However, tasks are synthetic (reverse-engineered from schemas), cover a limited set of domains (six), and may not capture the full messiness of real enterprise UIs or organizational heterogeneity (limitations).
SampleEntWorld comprises 1,756 tasks spanning six enterprise domains (including CRM, ITIL, and ERP), generated by reverse-engineering database schemas to synthesize realistic, long-horizon workflows; tasks are validated with a SQL-based deterministic verification mechanism; evaluated on state-of-the-art multimodal LLM agents (e.g., GPT-4.1) with reported agent success rate of 47.61% versus substantially higher human performance.
Themesproductivity adoption human_ai_collab org_design
GeneralizabilityTasks are synthetically generated from schemas and may not reflect real-world UI variability, latency, or noisy inputs., Coverage limited to six representative domains (CRM, ITIL, ERP, etc.) — other enterprise systems and industry-specific workflows may differ., Relies on availability of underlying database schemas and SQL-accessible state; many enterprises restrict schema access or use opaque systems., Evaluations focus on particular proprietary models (e.g., GPT-4.1) and may not generalize across model families or smaller on-premise models., Cultural, regulatory, and language/locale variations in enterprise workflows are not detailed and may limit cross-region applicability., Benchmark emphasizes structured state transitions and may under-represent unstructured content and multimodal UI complexities.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Recent advances in Multimodal Large Language Models (MLLMs) have enabled agents to operate in open-ended web and operating system environments. Other | positive | ability of agents to operate in open-ended web/OS environments |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Existing benchmarks predominantly target consumer-oriented scenarios (e.g., e-commerce and travel booking), failing to capture the complexity and rigor of professional enterprise workflows. Other | negative | coverage of benchmarks for enterprise workflows |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Enterprise systems pose distinct challenges, including high-density user interfaces, strict business logic constraints, and a strong reliance on precise, state-consistent information retrieval — settings in which current generalist agents often struggle. Output Quality | negative | agent performance in enterprise system settings |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We introduce EntWorld, a large-scale benchmark consisting of 1,756 tasks across six representative enterprise domains, including customer relationship management (CRM), information technology infrastructure library (ITIL), and enterprise resource planning (ERP) systems. Other | positive | benchmark size and domain coverage (number of tasks and domains) |
Reading fidelity
high
Study strength
high
|
n=1756
1,756 tasks
|
| Unlike previous datasets that depend on fragile execution traces or extensive manual annotation, EntWorld adopts a schema-grounded task generation framework that directly reverse-engineers business logic from underlying database schemas, enabling the synthesis of realistic, long-horizon workflows. Other | positive | realism and long-horizon workflow synthesis in dataset generation |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We propose a SQL-based deterministic verification mechanism in building datasets that replaces ambiguous visual matching with rigorous state-transition validation. Other | positive | dataset verification rigor (state-transition correctness) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Experimental results demonstrate that state-of-the-art models (e.g., GPT-4.1) achieve 47.61% success rate on EntWorld. Output Quality | negative | task success rate (correct task completion) |
Reading fidelity
high
Study strength
medium
|
n=1756
47.61%
|
| Model performance (47.61% for GPT-4.1) is substantially lower than human performance, highlighting a pronounced enterprise gap in current agentic capabilities. Output Quality | negative | gap between model and human task success rates |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We release EntWorld as a rigorous testbed to facilitate the development and evaluation of the next generation of enterprise-ready digital agents. Other | positive | availability of EntWorld as a public testbed |
Reading fidelity
high
Study strength
medium
|
not reported
|