← Papers

Giving agents direct, structured access to software state and semantic operations transforms desktop automation: across a 380-task benchmark ASIL raises agent success rates to ~80% while GUI-based screenshot-and-click control scores in the low double digits, and replaying verified ASIL traces for SFT/RL materially boosts weaker models.

ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

Rui Xie, Lu Chen · August 27, 2026

arxiv quasi_experimental medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 28, 2026

Rui Xie unresolved corpus identity
Lu Chen unresolved corpus identity

Semantic Scholar

Latest observation: August 28, 2026

Rui Xie provider ID
Lu Chen unresolved corpus identity

Replacing screenshot-and-click with a structured JSON observation and semantic, code-executable action interface (ASIL) yields much higher task success and far shorter action sequences for software-operating agents across 15 apps and 380 benchmark tasks, and ASIL-derived traces materially improve model performance via SFT and on-policy RL.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 28, 2026

View corpus context

Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.

Summary

Main Finding

ASIL (Agent–Software Interaction Layer) replaces the screenshot-and-click agent interface with an agent-native interface: structured JSON observations of software state and code-executable semantic actions. Across 15 real applications and a 380-task benchmark (300 single-app, 80 multi-app), ASIL yields large improvements in task success, action efficiency, and trainability versus conventional GUI (screenshot-and-click) agents. ASIL-enabled agents achieve >~80% strict success on the benchmark while executing on average fewer than five semantic actions per task; comparable screenshot-and-click runs score in the single- or low-double digits under realistic budgets.

Key Points

Interface shift: Replace pixel observations + low-level GUI events with structured OBSERVATION objects (task metadata, app state, interactive elements, etc.) and schema-constrained ACTION objects (modify_file, invoke_function, api_call, navigate, batch, etc.).
Realizations: Implemented via three practical access-path patterns — file-backed (structured files), native scripting runtimes, and service/API endpoints — normalized into a single JSON contract per app.
Implementation & coverage: ASIL instantiated across 15 applications (creative tools, productivity suites, code/service tools, desktop utilities) with a semi-automatic onboarding pipeline for adapter/profile generation and verification.
Benchmark: Single evaluation system of 380 tasks that reuses the same initial states and validators for ASIL and GUI runs; ASIL renders per-step GUI snapshots from internal state so visual inspection and direct comparisons are possible.
Performance:
- Typical result examples: GPT-5.4 under ASIL ≈ 81.6% strict success vs GUI ≈ 6.6% under a repaired 50-step GUI budget; other models (e.g., sonnet4.6) show similarly large ASIL–GUI gaps (ASIL ≈ 81, GUI varying with budget).
- ASIL average executed actions per task < 5 (default ASIL budget 15 steps).
- On an “easier” OSWorld-comparable subset GUI performance improves but still trails ASIL substantially.
Native API comparisons: On 60 LibreOffice tasks, ASIL outperforms the native UNO API by 28–38 strict points; on 20 draw.io tasks ASIL matches the draw.io MCP content contract for some models but may trail for others — i.e., ASIL is compositional and often easier for agents to use than some low-level native APIs.
Training benefits:
- Verified ASIL trajectories are reusable for SFT and on-policy RL.
- Small SFT gains: Qwen3.5-2B from 58.0 → 72.1; Qwen3.5-9B from 66.6 → 80.4.
- Adding resource-limited on-policy RL yields further gains: 2B → 74.4, 9B → 82.2.
Robust evaluation loop: The same evaluator (final-state checks) is used for inference, SFT filtering, and RL rewards, improving replayability and verification.

Data & Methods

Benchmark:
- 380 tasks: 300 single-application tasks (15 apps × 20 each) + 80 multi-application workflows that require cross-app artifact/state movement.
- Shared task definitions, initial artifacts, and software-aware validators used for both ASIL and GUI modes.
- ASIL default budget: 15 semantic-action steps; repaired GUI baseline budget: up to 50 GUI events (with additional 15-step truncated comparisons).
Application realization:
- Three realization patterns: file-backed execution (e.g., SVG, ODF), native scripting (e.g., Blender Python), and service/API calls (REST/WebSocket).
- Semi-automatic onboarding: human-reviewed interface profiles compiled into adapter contracts; automated probes and validation checks used (example: GPT-5.4 generated a 97-line Gitea API profile quickly with zero audit errors).
Training pipeline:
- Verified ASIL rollouts and expert (GPT-5.4) rollouts provide step-level SFT traces (SFT-v0 + guided-v2) and the replay buffer for RL.
- On-policy RL performed via an ASIL AgentService; rewards computed by the same state-aware evaluator used for scoring.
- Training pools: separate low-overlap training task pools (e.g., 320/80 RL subset drawn from a final 512/128 v3 pool) to avoid evaluation leakage.
Key metrics reported:
- Strict success rates per-model and per-interface (ASIL vs GUI), average number of actions, and training deltas after SFT and RL.
- Matched native-interface baselines (LibreOffice UNO, draw.io MCP) used for validity checks.

Implications for AI Economics

Reduced operating costs per task:
- Fewer agent steps (semantic actions vs long GUI action chains) mean fewer model invocations, lower per-task latency, and reduced cloud inference costs (token/model-call fees and wall-clock usage). ASIL average <5 actions vs GUI often tens-to-hundreds of steps.
- Eliminates repeated heavy multimodal processing (screenshot acquisition + vision inference) per step, lowering compute and I/O expenditure.
Lower training/sample costs and better sample efficiency:
- Shorter, higher-signal trajectories shrink RL credit-assignment horizons and reduce rollout costs (less environment time, fewer validator calls).
- Verified, software-aware traces improve SFT data quality — leading to large accuracy gains with modest SFT scale and modest extra RL, reducing the need for massive, noisy human-labeled datasets.
- Reusable, replayable traces and a unified evaluator reduce waste in experimentation and hyperparameter search.
Faster time-to-productivity and deployment:
- ASIL’s semantic actions are more robust to UI changes (layout/theme) than pixel-level control, reducing maintenance and re-training costs for deployed agents.
- Semi-automated onboarding that maps to deepest feasible access path allows faster integration for many applications compared to building brittle GUI wrappers.
Strategic implications for software/platform owners:
- Incentive to expose stable programmatic surfaces (APIs, scripting, structured formats) and standardized state contracts to enable efficient agent automation; platforms that do so may capture more workflow automation value.
- Firms that only allow GUI access may see lower automation quality from third-party agents, potentially motivating providers to offer agent-friendly interfaces (new revenue channels or platform lock-in).
Market shifts and productivity effects:
- Large gains in per-task automation reliability and speed could accelerate adoption of software-operating agents in office, creative, and developer tooling — increasing labor substitution/augmentation effects and changing demand for tasks that depend on manual GUI manipulation.
- Cost-per-automation decreases as ASIL reduces compute and engineering overhead, making many micro-automation tasks economically viable.
Remaining costs and limits:
- Onboarding and adapter development: although semi-automated, building and validating adapters (deep access paths) across many closed/proprietary apps still incurs engineering and auditing costs.
- Coverage limitations: opaque or closed apps with no parseable file/scripting/service surface remain out-of-scope; such apps require continued investment or vendor cooperation.
- Security, compliance, and governance costs: deeper access to app state and programmatic actions raises new policy and security needs (access control, data leakage prevention, audit trails), which impose operational overhead.
Net economic takeaway:
- ASIL materially reduces the marginal cost (compute + time + failure risk) of automating GUI-heavy workflows and materially increases the productivity/quality of automation. This lowers barriers to large-scale practical deployment of software-operating agents, shifts incentives toward exposing structured interfaces, and changes where value accrues in the automation stack — but nontrivial upfront adapter/custody and governance costs remain.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The paper reports large, consistent performance gaps (e.g., ~80% vs single-digit GUI scores) across a broad, implemented 15-app, 380-task benchmark and supplements main comparisons with repaired-GUI bands and native-API baselines; it also demonstrates training gains from verified ASIL traces. However, causal claims are qualified by residual confounds (prompt/hint differences in some arms, non-randomized comparisons, reliance on closed/proprietary models for headline numbers, and potential task-selection/design effects), which limit the strength of causal inference. Methods Rigormedium — The authors produce a broad, reproducible benchmark with shared validators, renderings, and replayable traces; they implement multiple realization patterns (file/script/API) across 15 applications and run ablations and matched native baselines. Yet some methodological choices weaken rigor: the main comparison admits prompt/hint differences (later addressed in additional arms), runs include closed models (reducing reproducibility), there is no randomized trial or counterfactual assignment of tasks/models, and documentation of statistical uncertainty and failure modes is limited in the supplied text. SampleImplemented across 15 desktop and service-backed applications (e.g., Blender, Inkscape, GIMP, Audacity, Kdenlive, LibreOffice Calc/Writer/Impress, draw.io, Jupyter/code-server, Gitea, Thunderbird, Nautilus, OBS). Benchmark: 300 single-application tasks (20 per app) + 80 multi-application workflows = 380 tasks total. ASIL realized via file-backed, native-script, and service/API adapters. Evaluated models include closed models (GPT-5.4) and open/closed family models (Qwen3.6+, Qwen3.5-2B/9B, sonnet4.6, etc.). Default run budgets: ASIL 15-step (avg <5 actions per task); repaired GUI comparisons use up to 50-step budgets (also 15-step truncations for parity). Training: SFT traces derived from verified ASIL rollouts; on-policy RL uses evaluator-backed rollouts from a 512/128 (v3) training pool and smaller RL subsets (320/80). Themesproductivity human_ai_collab IdentificationWithin-benchmark, same-task comparisons: the authors implement a single 380-task benchmark (300 single-app + 80 multi-app) and run agents in two interface modes (ASIL: structured JSON observations + semantic actions; GUI: screenshot-and-click). Tasks, initial artifacts, and final-state validators are shared across modes so performance is compared on identical task definitions; additional ablations include repaired GUI budgets, hint-off comparisons, and matched native-API baselines (LibreOffice UNO, draw.io MCP). They also evaluate training effects by replaying verified ASIL traces for SFT and using evaluator-backed on-policy RL. No randomized assignment or external instrumental variables are reported; control is achieved by matched tasks, shared validators, and targeted ablations to probe confounds (e.g., hint effects, GUI budget). GeneralizabilityExcludes applications without parseable file/script/API access paths (opaque closed GUIs remain out of scope)., Results depend on availability and quality of access paths/adapters; many commercial desktop/web apps may not expose suitable surfaces., Headline results rely partly on closed large models (reducing reproducibility and raising uncertainty about replication on other models)., Benchmark task selection and validator definitions may favor semantic, state-centered interventions (ASIL) over pixel-based recovery tasks., Operating-system, deployment, and real-world latencies/permissions when integrating adapters are not fully captured and may affect field applicability., Training gains reported for SFT/RL are resource- and dataset-dependent and may not scale identically to very different models or task pools.

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
On the 380-task benchmark, GPT-5.4 achieves an overall score of 81.6 with ASIL versus 6.6 with repaired screenshot-and-click GUI control. Task Completion Time	positive	Strict task success score	Reading fidelity high Study strength medium	n=380 81.6 versus 6.6 strict points 0.48
On the 380-task benchmark, sonnet4.6 achieves 81.2 with ASIL versus 26.6 with repaired screenshot-and-click GUI control, and 17.9 when restricted to 15 GUI actions. Task Completion Time	positive	Strict task success score	Reading fidelity high Study strength medium	n=380 81.2 versus 26.6 strict points; 17.9 at 15 GUI actions 0.48
ASIL agents complete benchmark tasks with fewer than five executed actions on average under the default 15-step budget. Organizational Efficiency	positive	Number of executed actions per task	Reading fidelity high Study strength medium	n=380 fewer than five actions per task 0.48
On an easier 60-task single-application band, repaired 50-step GUI performance increases to 15.0 strict points for GPT-5.4 and 53.3 strict points for sonnet4.6. Task Completion Time	positive	Strict task success score on an easier task band	Reading fidelity high Study strength medium	n=60 15.0 and 53.3 strict points 0.48
On 60 LibreOffice tasks, ASIL exceeds the LibreOffice UNO API baseline by 28–38 strict points. Organizational Efficiency	positive	Strict task success score	Reading fidelity high Study strength medium	n=60 28–38 strict points 0.48
On 20 draw.io tasks, ASIL matches draw.io’s MCP content contract for GPT-5.4 but performs worse than it for sonnet4.6. Organizational Efficiency	mixed	Strict task success score	Reading fidelity high Study strength medium	n=20 0.48
Small-scale supervised fine-tuning improves Qwen3.5-2B’s score from 58.0 to 72.1 and Qwen3.5-9B’s score from 66.6 to 80.4 under the ASIL modality. Training Effectiveness	positive	Benchmark task success score after supervised fine-tuning	Reading fidelity high Study strength medium	n=380 58.0 to 72.1 for Qwen3.5-2B; 66.6 to 80.4 for Qwen3.5-9B 0.48
Resource-limited on-policy reinforcement learning further improves Qwen3.5-2B from 72.1 to 74.4 and Qwen3.5-9B from 80.4 to 82.2. Training Effectiveness	positive	Benchmark task success score after reinforcement learning	Reading fidelity high Study strength medium	n=380 72.1 to 74.4 for Qwen3.5-2B; 80.4 to 82.2 for Qwen3.5-9B 0.48
The ASIL benchmark covers 15 software applications with 300 single-application tasks and 80 multi-application tasks, for 380 tasks in total. Other	positive	Benchmark coverage and task count	Reading fidelity high Study strength high	n=380 300 single-application tasks plus 80 multi-application tasks 0.8