A new 100-task benchmark finds LLM agents can complete long office-suite jobs far faster and cheaper than humans, but still underperform on deliverable quality: the best model scores 17.9 versus a human baseline of 27.8 on OmegaUse-OfficeVal.

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu · July 29, 2026

arxiv descriptive n/a evidence 8/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 4, 2026

Jingbo Zhou unresolved corpus identity
Yusai Zhao unresolved corpus identity
Qi Bao unresolved corpus identity
Jingjia Cao unresolved corpus identity
Zhenghai Chen unresolved corpus identity
Chang Gao unresolved corpus identity
Kaiqi Guo unresolved corpus identity
Muxin Guo unresolved corpus identity
Mingxuan Li unresolved corpus identity
Xinjiang Lu unresolved corpus identity
Yanru Ma unresolved corpus identity
Yixiong Xiao unresolved corpus identity
Zenghui Zhang unresolved corpus identity
Le Zhang unresolved corpus identity
Hua Wu unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

Jingbo Zhou provider ID
Yusai Zhao provider ID
Qi Bao provider ID
Jingjia Cao provider ID
Zhenghai Chen provider ID
Chang Gao provider ID
Kai Guo provider ID
Muxin Guo provider ID
M. Li provider ID
Xinjiang Lu provider ID
Yanru Ma provider ID
Yixiong Xiao provider ID
Zenghui Zhang provider ID
Le Zhang provider ID
Hua Wu provider ID

OmegaUse-OfficeVal is a 100-task benchmark of long-horizon office-suite workflows with per-task human labor-time and price-proxy annotations, showing that current LLM agents are substantially faster and cheaper than humans but still produce lower-quality deliverables.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.

Summary

Main Finding

OmegaUse-OfficeVal is a new, fully open benchmark of 100 long-horizon office-suite workflows that pairs per-task economic signals (human labor time and a task price proxy) with code-based, fine-grained verifiers. Evaluations show frontier LLM agents run substantially faster and cheaper than humans but still lag materially in final deliverable quality (best model score 17.91 vs. human 27.79). The benchmark enables direct, value-aware comparisons of LLM vs. human productivity for realistic office work.

Key Points

Dataset scope
- 100 long-horizon office-suite tasks (DOCX, PPTX, XLSX, PDF, images, videos).
- 220 input files and 115 expected output artifacts.
- Average human labor time per task: 2.32 hours.
- Average rubric granularity: ~20.09 rubric items per task.
Economic grounding
- Two per-task economic signals: human labor time (measured via incentivized annotators) and a task price proxy (explicit market price when available; otherwise aggregated expert estimates).
- Prices reported in CNY (paper converts to USD with 1 CNY = 0.14609 USD).
Evaluation design
- Focus on final deliverable quality/usability rather than execution trajectory (GUI actions versus artifact).
- Stable, reproducible scoring via code-based verifiers derived from fine-grained rubrics; process includes expert revision and human–code discrepancy resolution.
- Value-weighted evaluation is supported (so economically important tasks can be prioritized/weighted).
Empirical results
- Models evaluated: GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, Qwen3.7-Plus, plus human baseline.
- Best LLM score: 17.91; human baseline: 27.79.
- All evaluated LLMs are faster and cheaper than humans, but quality gap remains significant.
- Performance varies with task difficulty/human labor time; highest average model score need not map to highest captured economic value.
Openness
- Complete assets released (instructions, input files, rubrics, verifier code, economic annotations).
- Project site: https://omegause-officeval.github.io

Data & Methods

Task collection
- Derived from authentic practitioner requests, adapted with privacy-preserving procedures.
- Tasks are high-level user instructions plus multimodal input files; deliverable is a completed office artifact.
Economic signal construction
- Human labor time: recorded from human annotators completing tasks without LLM assistance under an incentive mechanism (details in appendix).
- Task price proxy: either direct practitioner price or consistency-based aggregation of expert estimates when explicit price absent.
Verification & scoring
- Fine-grained rubrics generated per task, reviewed by experts (average ~20 rubric items/task).
- Rubrics translated into executable verification code; iterative human–code discrepancy resolution used to align code and human judgment.
- Rubrics both reward required items and penalize unintended damage (repair burden).
Experimental setup
- Agents evaluated by producing the final artifact; verifiers assign scores and record runtime and inference cost.
- Metrics: deliverable quality score, runtime, monetary cost; also value-weighted aggregate metrics.
- Detailed execution environment, prompts, and inference config provided in appendices.
Illustrative task heterogeneity (examples from paper)
- Reformatting and templating Word reports — price proxy $7.01, human time 287 min.
- Create editable answer-sheet from exam paper — $8.77, 92 min.
- Add animated videos and layout PPT changes — $3.65, 43 min.
- Build Excel macro to select area and open print preview — $29.22, 344 min.
- Recreate flow diagram as editable slide — $5.84, 239 min.
- Plot multiple financial time-series charts in Excel — $4.38, 93 min.

Implications for AI Economics

Enables direct ROI-style comparisons
- Per-task human labor time + price proxy allow computation of human cost vs. LLM inference cost and value-capture metrics for specific office tasks.
- Facilitates empirical estimates of where LLM agents already generate positive economic returns and where gaps remain.
Granular labor-substitution insights
- Heterogeneous task times and values show substitution effects will be uneven: short/low-value tasks may be automated sooner, while long-horizon, high-value tasks still need quality improvements.
- Value-weighted evaluation highlights that maximizing average model score is not identical to maximizing economic impact—models should be optimized for value capture.
Importance of deliverable quality and repair costs
- Lower-quality LLM outputs impose repair burden on users; rubric design explicitly penalizes unintended damage, underscoring that cost comparisons must include downstream repair/validation labor.
Marketplace and pricing design
- Task-level price proxies can inform product pricing, human-in-the-loop workflows, and deciding when to route a job to an LLM vs. a human worker.
- Benchmark can be used by firms to prioritize automation efforts where expected net benefit (time & cost saved minus repair/validation) is largest.
Policy and labor-market considerations
- Evidence that LLMs are cheaper/faster but lower quality suggests partial automation and task reallocation rather than immediate wholesale displacement.
- Call for more empirical studies using task-level economic signals to model employment impacts and reskilling needs.
Research directions motivated by economic goals
- Improve long-horizon planning, cross-file consistency, and artifact usability to close quality gaps.
- Integrate cost-aware training/objectives (optimize for value-weighted metrics, reduce repair-prone behaviors).
- Extend benchmarks to include dynamic pricing, multi-step human-in-the-loop costs, and longitudinal productivity studies.

If you want, I can: - Extract the full per-task table with human-times and price proxies for quick ROI calculations. - Draft an example calculation comparing human cost vs. LLM inference cost (including repair overhead) for one or more tasks.

Assessment

Paper Typedescriptive Evidence Strengthn/a — This is a benchmark and evaluation paper, not a causal-inference study; it reports descriptive empirical performance and cost comparisons rather than identifying causal effects. Methods Rigorhigh — Systematic task collection from practitioner requests, per-task economic annotations (human labor time and price proxy), an explicit incentive protocol for time measurement, fine-grained rubrics translated into executable verifiers, and evaluation against multiple frontier LLMs plus a human baseline; limitations include potential selection/coverage bias in task sourcing, reliance on expert-aggregated price proxies, and possible gaps between code-based verification and real-world subjective usability. SampleA benchmark of 100 long-horizon office-suite tasks derived from authentic practitioner requests (privacy-preserved), with 220 input files across modalities (DOCX, PPTX, XLSX, PDF, images, videos); tasks average 2.32 hours of human labor; each task annotated with human labor time and a task price proxy (reported in CNY and converted to USD); fine-grained rubrics and executable code-based verifiers accompany each task; evaluated several LLMs (GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, Qwen3.7-Plus) and a human baseline. Themesproductivity human_ai_collab GeneralizabilityTasks focus narrowly on office-suite workflows (documents, spreadsheets, presentations, PDFs) and may not generalize to other types of work., Tasks were sourced from a specific practitioner pool (and processed for privacy), which may introduce selection bias and limit representativeness across industries and geographies., Economic annotations (price proxies) rely on explicit practitioner prices when available or on expert aggregation, which may not reflect market variability or different labor markets outside the sample., Code-based verifiers capture objective rubric criteria but may miss subjective aspects of usability, style, or domain-specific nuance that affect real-world adoption., Evaluated LLMs represent a subset of available models and configurations; results may change with different agent architectures, tool chains, or richer integration with GUI automation.

Claims (10)

Claim	Direction	Outcome	Confidence & Evidence	Details
OmegaUse-OfficeVal comprises 100 long-horizon office-suite tasks derived from authentic office requests proposed by practitioners and adapted through a privacy-preserving process. Organizational Efficiency	null_result	Benchmark task coverage and authenticity	Reading fidelity high Study strength medium	n=100 0.18
The benchmark tasks require an average of 2.32 hours of human labor to complete. Task Completion Time	null_result	Human labor time required per task	Reading fidelity high Study strength medium	n=100 2.32 hours per task 0.18
Every benchmark task is paired with two economic signals: human labor time and a task price proxy. Organizational Efficiency	positive	Availability of task-level economic value annotations	Reading fidelity high Study strength medium	n=100 0.18
The benchmark uses executable code-based verifiers derived from fine-grained rubrics rather than human or LLM-as-judge scoring. Ai Safety And Ethics	positive	Evaluation reproducibility and scoring stability	Reading fidelity high Study strength medium	n=100 0.18
The best evaluated LLM agent achieved a benchmark score of 17.91, compared with 27.79 for the human baseline. Output Quality	negative	Final office-deliverable quality and task score	Reading fidelity high Study strength medium	n=100 17.91 for the best LLM versus 27.79 for humans 0.18
The evaluated LLM agents are cheaper than human workers for the benchmark tasks. Organizational Efficiency	positive	Cost of completing office-suite tasks	Reading fidelity high Study strength medium	n=100 0.18
The evaluated LLM agents are faster than human workers for the benchmark tasks. Task Completion Time	positive	Task completion time	Reading fidelity high Study strength medium	n=100 0.18
Despite being cheaper and faster, the evaluated LLM agents have not yet reached human-level quality for final office deliverables. Output Quality	mixed	Trade-off between deliverable quality, cost, and completion time	Reading fidelity high Study strength medium	n=100 Best LLM score 17.91 versus human score 27.79 0.18
The model with the highest average score is not necessarily the model that captures the most economically important tasks under value-weighted evaluation. Organizational Efficiency	mixed	Economic value captured by agent performance	Reading fidelity high Study strength medium	n=100 0.18
OmegaUse-OfficeVal contains 220 input files and 115 required output artifacts. Organizational Efficiency	null_result	Benchmark input and output artifact counts	Reading fidelity high Study strength high	n=100 220 input files; 115 output artifacts 0.3