LabOSBench offers a low-cost, browser-based simulation benchmark for scientific-instrument control across 96 subtasks; existing multimodal GUI agents handle structured actions but fail to reliably perform feedback-driven tuning and extended workflows, revealing gaps before real-world lab automation.

LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

Anqi Zou, Han Deng, Chengyu Zhang, Junquan Hu, Yu Wang, Yuxiang Xing, Aokai Zhang, Hanling Zhang, Zhaoyang Liu, Ben Fei, Zhihui Wang, Wanli Ouyang · June 15, 2026

arxiv descriptive n/a evidence 7/10 relevance Full text usable extracted full text Source PDF

LabOSBench is a browser-based benchmark of eight simulated scientific instruments (96 subtasks) that shows current multimodal GUI agents can complete many structured operations but struggle with feedback-driven parameter tuning and long-horizon experimental workflows.

Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment. However, directly evaluating agents on physical high-precision instruments is impractical due to high cost, safety risks, limited accessibility, and difficulty in ensuring reproducible evaluation. This motivates the need for a simulated yet realistic testbed that preserves the operational challenges of scientific instruments while enabling scalable and safe benchmarking. To this end, we introduce LabOSBench, a challenging benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators. Operating directly via a browser, LabOSBench avoids resource-heavy OS virtualization while supporting flexible task configuration and execution-based evaluation. Specifically, LabOSBench constructs 96 subtasks across eight instrument simulators, covering workflows from sample loading, alignment, parameter tuning, and data acquisition to result inspection. We evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels. Our experiments reveal that while existing agents can complete many structured GUI subtasks, they still struggle with feedback-driven operations and long-horizon workflow execution. Overall, LabOSBench provides a reproducible, low-cost testbed for advancing computer-using agents toward scientific-instrument control.

Summary

Main Finding

LabOSBench introduces a lightweight, browser-based benchmark of scientific-instrument GUI control that enables reproducible, low-cost evaluation of multimodal computer-use agents. Across 96 subtasks spanning eight instrument simulators, current state-of-the-art multimodal LLM/VLMs and agentic frameworks can handle many structured GUI actions but systematically underperform on feedback-driven parameter tuning, scientific-state interpretation, action localization, and long-horizon workflow recovery. Agentic planning frameworks improve overall performance but remain far from human-level reliability.

Key Points

Purpose: Fill a gap between web/OS benchmarks and real instrumentation by providing realistic, executable GUI simulators for scientific instruments where device APIs are unavailable or impractical.
Scope: 8 instrument simulators (SEM, TEM, LFM, XRD, EDS, APT, FIB, SPM) and 96 subtasks covering sample prep, environment conditioning, activation/alignment, parameter configuration, acquisition, and post-processing.
Evaluation modes:
- Full-episode (long-horizon workflows from initial state).
- Subtask-level (simulator fast-forward to the pre-condition of a subtask) for diagnostic isolation.
Action space: clicks, drags, typing, dropdown selections, keypresses, scrolling, waiting; executed in-browser via Playwright with DOM events where possible.
Metrics: subtask-normalized success rates, episode success (all subtasks done within step budget), plus domain-specific image quality metrics (e.g., PSNR for SEM micrographs).
Representative empirical findings:
- Best non-human performer: GTA1 w/ GPT-5.5 (avg subtask score ≈ 0.814).
- Strong general models: GPT-5.5 (≈ 0.726), Seed-1.6 (≈ 0.763).
- Human baseline: ≈ 0.924 average.
- Performance gaps largest on FIB (long workflows, error accumulation) and LFM (feedback-driven optical adjustments).
Failure modes identified: poor closed-loop feedback adjustment, inadequate scientific-state interpretation, visual grounding/localization errors, poor recovery from early mistakes.

Data & Methods

Implementation:
- Web-based interactive simulators (no heavy OS virtualization) instrumented with in-page benchmark scripts that export episode logs and subtask completion traces via JSON schemas.
- Browser driver (Playwright) captures screenshots and executes agent actions; step traces and screenshots saved for analysis.
Task decomposition:
- Workflows decomposed into human-meaningful subtasks (not arbitrary screen regions); subtasks correspond to domain-relevant operations.
- Fast-forward functions used in subtask evaluation to initialize the UI to a subtask start state.
Benchmarked models:
- General multimodal VLMs: Qwen3VL-32B, EvoCUA-8B, Claude Sonnet-4.5, Kimi-K2.5, Seed-1.6, GPT-5.5, Claude Opus-4.5.
- Specialized GUI models: UI-TARS-1.5-7B, GUI-Owl-7B.
- Agentic frameworks: GTA1 (w/ GPT-5.5), VLAA-GUI (w/ Opus-4.5), Hippo Agent (w/ Opus-4.5).
Evaluation protocol:
- Subtask-level: up to 50 interaction steps per subtask; Rd,s = 2 runs per model–subtask pair.
- Aggregation: subtask-normalized instrument scores; macro-average across instruments for benchmark-level score.
Quantitative summary (selected numbers):
- Avg. subtask scores (examples): GTA1 w/ GPT-5.5 = 0.814; GPT-5.5 = 0.726; Seed-1.6 = 0.763; UI-TARS-1.5-7B = 0.659; human = 0.924.
- Instruments with notable difficulty: FIB (longest workflows, lowest model performance), LFM (high human–model gap for feedback-driven optics).

Implications for AI Economics

Lowering evaluation cost and entry barriers
- Browser-based simulators eliminate heavy VM infrastructure, reducing fixed costs of benchmarking and enabling more frequent, scalable model iteration. This reduces the cost of experimentation for startups and labs and accelerates the pace of model development targeted at lab automation.
Market implications and demand signals
- Demonstrated gaps in feedback-driven, closed-loop control suggest high economic value for specialized models and agentic frameworks tailored to instrument-specific state estimation and recovery. Expect growing demand (and willingness to pay) for domain-adapted multimodal agents, instrument-specific adapters, and polished integration tooling.
- Instrument vendors may find commercial opportunities to offer higher-fidelity simulators, APIs, or certified control stacks (reducing generalization risk) — shifting revenue toward software and services around hardware.
Returns to capital vs. labor in R&D
- If multimodal agents can reliably control instruments, marginal costs of routine experimental runs fall (fewer skilled-operator hours required), increasing capital productivity. This implies potential substitution of some technician tasks with software capital, altering wage-pressure and skill demand toward higher-level experiment design, validation, and exception handling.
- However, current performance gaps (human ≈ 0.924 vs. best agents ≈ 0.814) mean near-term complementarity: agents increase throughput for structured tasks but human oversight remains critical for feedback-heavy procedures.
Risk, safety, and regulation externalities
- Simulators reduce physical risk during development/testing, but real-world deployment raises safety, liability, and regulatory concerns (e.g., instrument damage, hazardous materials). Economic models of adoption should incorporate costs of certification, insurance, and liability mitigation.
Investment and infrastructure needs
- Growth areas: high-fidelity simulators, domain-specific fine-tuning datasets, state-estimation modules, automated recovery strategies, and datasets for closed-loop control. Investors may favor platforms bundling models with instrument virtualization and compliance tooling.
- Public goods and standardization: open benchmarks like LabOSBench reduce information asymmetry and foster competition, accelerating technological diffusion but also making it easier for lower-cost entrants to compete in lab automation services.
Research and policy directions for economists
- Quantify productivity gains: measure changes in cost-per-experiment, throughput, and time-to-discovery with agent-assisted instruments.
- Labor-market effects: project task reallocation within labs (which tasks are automated vs. augmented) and estimate wage impacts for technical staff.
- Diffusion dynamics: model adoption curves accounting for reliability thresholds, regulatory overhead, and complementarities with digital infrastructure.
- Market structure: analyze potential platform monopolies if major model vendors or instrument manufacturers vertically integrate models + hardware APIs.
Caveats & generalization risks
- Simulator-to-real gap: economic conclusions must account for transfer risk; solving GUI tasks in simulation does not guarantee safe, robust real-device control.
- Heterogeneous returns: benefits will vary by domain (high-throughput, structured assays easier to automate; delicate, feedback-heavy work harder).
- Potential negative externalities: accelerated experiment throughput could increase hazardous waste or dual-use risks; regulation and governance may be needed.

Suggested empirical metrics for economic studies using LabOSBench - Cost-per-evaluation (compute + human oversight) and marginal cost reductions from simulator-driven development. - Calibration of model reliability thresholds at which firms substitute labor with agentic control. - Investment-to-productivity ratios for firms adopting instrument-control agents (capex + software vs. incremental throughput). - Price elasticity for instrument software/services post-introduction of domain-specific agent capabilities.

Overall, LabOSBench is an enabling infrastructure that materially reduces the cost of benchmarking and iterating instrument-control agents, which in turn can accelerate market development for lab-automation software, change the capital–labor mix in experimental science, and create both commercial opportunities and governance challenges.

Assessment

Paper Typedescriptive Evidence Strengthn/a — Paper introduces and evaluates a benchmark (LabOSBench) rather than testing causal hypotheses about economic outcomes; results describe agent capabilities within the simulated testbed but do not establish causal effects on productivity or labor. Methods Rigormedium — The authors build a substantial, systematic benchmark (eight web-based instrument simulators, 96 subtasks) and evaluate multiple model families at subtask and end-to-end levels, which shows careful design and breadth; however, the work lacks validation on real physical instruments, may have selection bias in task design, and does not demonstrate robustness across diverse hardware or lab contexts. SampleA web-browser-based suite of eight simulated scientific instruments implementing 96 subtasks spanning sample loading, alignment, parameter tuning, data acquisition, and result inspection; evaluated using several general-purpose vision-language models, specialized GUI agent models, and agentic frameworks, with execution-based metrics reported at both subtask and end-to-end workflow levels. Themesproductivity human_ai_collab GeneralizabilitySimulated instruments may not capture physical dynamics, noise, safety constraints, or hardware failure modes of real lab equipment, Limited to eight instrument types and 96 configured subtasks, so domain coverage across all scientific instrumentation is narrow, Browser-based GUI interactions omit OS-level and hardware-specific interfaces common in real labs, Task and metric design choices could bias evaluation toward particular agent architectures, Models evaluated are a snapshot and may not represent future or proprietary agent designs

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems. Other	null_result	benchmark_scope	Reading fidelity high Study strength medium	not reported 0.18
Scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment. Other	null_result	requirements_of_instrument_control	Reading fidelity high Study strength low	not reported 0.09
Directly evaluating agents on physical high-precision instruments is impractical due to high cost, safety risks, limited accessibility, and difficulty in ensuring reproducible evaluation. Other	negative	practicality_of_physical_evaluation	Reading fidelity high Study strength medium	not reported 0.18
LabOSBench is a challenging benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators that operates directly via a browser and thus avoids resource-heavy OS virtualization while supporting flexible task configuration and execution-based evaluation. Other	positive	benchmark_capabilities	Reading fidelity high Study strength medium	not reported 0.18
LabOSBench constructs 96 subtasks across eight instrument simulators, covering workflows from sample loading, alignment, parameter tuning, and data acquisition to result inspection. Other	null_result	benchmark_scope (number_of_subtasks, number_of_simulators)	Reading fidelity high Study strength high	n=96 8 instrument simulators 0.3
The authors evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels on LabOSBench. Other	null_result	evaluation_coverage (model_types and evaluation_levels)	Reading fidelity high Study strength medium	not reported 0.18
Existing agents can complete many structured GUI subtasks. Task Completion Time	positive	subtask_completion_success	Reading fidelity medium Study strength medium	not reported 0.11
Existing agents still struggle with feedback-driven operations and long-horizon workflow execution. Task Completion Time	negative	performance_on_feedback_driven_and_long_horizon_tasks	Reading fidelity medium Study strength medium	not reported 0.11
LabOSBench provides a reproducible, low-cost testbed for advancing computer-using agents toward scientific-instrument control. Other	positive	testbed_reproducibility_and_cost	Reading fidelity high Study strength medium	not reported 0.18