Reallocating a fixed number of archived screenshots to distant, task-relevant events boosts GUI agent success: CausalCache raises zero-shot MobileWorld success by 3.7 percentage points overall and by 8.6 points on memory-critical tasks at the same visual budget, while leaving controls and desktop performance unchanged.

Main Finding

CausalCache improves long-horizon GUI agent performance by selectively restoring archived screenshots (high-fidelity images) for a subset B of past events, reallocating the same visual budget away from a strict recency heuristic (Recent-B). A policy-preserving history-gated KV adapter (HGKV) plus a budget-aware selector produces meaningful gains where memory matters (zero-shot across platforms), while keeping the base policy intact and holding the total visual budget constant.

Key Points

• Problem reframing: treat GUI memory as conditional fidelity restoration — every event stays in summary form, and only up to B events are promoted back to summary+image; the decision is which events to promote, not whether to store them.
• Baseline vs CausalCache: Recent-B always promotes the B most recent events. CausalCache reallocates those B slots over the whole history, replacing a recent image only if a distant event yields higher conditional marginal utility to the (frozen) policy.
• Policy-preserving adapter (HGKV): low-rank residuals (rank 8, α=16) are added to K and V projections in the last 8 transformer layers, gated so they act only on restored history-image tokens. With B=0 the adapter is structurally bypassed and the policy is bitwise-identical to the frozen base.
• Training objective: matched-budget replacement groups and a per-arm anchored difference-in-differences (DiD) loss ensure the adapter learns selective utility of specific restored images while preventing uniform amplification or drift. Loss parameters: m=0.01, ϵ=0.02, λcap=2, λ=1e-4.
• Selector and decision flow: two-pass procedure — pass 1 uses Recent-B to produce an action proposal; a budget-aware scorer conditioned on that proposal ranks candidate restorations and returns an exact-B set S. If S displaces recent items, a second pass renders the prompt with S* and emits the final action. Selector is trained on true policy utilities computed via the frozen HGKV policy.
• Training data and setup: adapter and selector trained only on desktop AgentNet/OpenCUA Ubuntu successful trajectories (5k screened → 2,293 successful → 6,003 decision points). Matched-budget groups: B=1/2/4 with 969/764/470 groups; held-out test groups and dev split used for checkpointing. Model: frozen GUI-Owl-1.5-8B-Instruct; all training and serving on NVIDIA H200.
• Empirical gains (B=4 primary):
  • MobileWorld zero-shot: overall success +3.7 percentage points (30.2 → 33.9).
  • Memory-critical subset: +8.6 points (19.4 → 28.0).
  • Single-app control subset: no improvement (−1.8), indicating gain concentrates where memory is needed.
  • OSWorld-Verified (desktop zero-shot): no reliable improvement (−0.3), consistent with the frozen policy already being similarly indifferent to archived vs recent frames in that domain.
• Practical design choices: training focuses on single-slot (k=1) replacements since multi-frame demand was rare in mining; deployment can compose multi-slot reallocations via the selector.

Data & Methods

• Problem formalization:
  • At decision t, history Ht = {e1...et−1} where each ej has a low-fidelity one-line summary ˜ej and an archived high-fidelity pair (rj, vj). Active visual-context budget B defines |S| = min(B, t−1) events shown with images.
  • Objective is to choose S maximizing the policy’s mean target-token log-likelihood Q(S) under a fixed policy and prompt structure; contrasts are same-B allocations to isolate fidelity allocation from visual capacity.
• Adapter (HGKV):
  • Adds ∆W^K_l and ∆W^V_l low-rank residuals in last 8 layers; multiplied by a binary token mask Mhist that is 1 only on promoted history-image tokens. If no history images are promoted, adapter is effectively removed (bitwise-equal behavior).
• Per-B supervision:
  • Construct matched-budget triplets: Recent-B, Relevant-restore (swap oldest recent slot for the earlier true target screenshot), Wrong-restore (age-matched but non-informative replacement).
  • DiD anchoring: compute As, Ar, An as differences between adapter-on and adapter-off log-likelihoods per arm, and optimize a hinge-style DiD objective plus drift caps.
• Selector:
  • Trained to predict set-conditional marginal utilities b∆t(j | S) using joint regression + within-state ranking loss. Deployed with beam search initialized at Recent-B and recent fallback — a distant event must beat the marginal of the recent item it would evict.
  • Uses the policy’s own proposal as a reference during training (proposal-conditioned rescoring), since the gold post-action oracle is unavailable at deployment.
• Evaluation:
  • Offline: teacher-forced log-likelihood margins, action-equivalence metrics.
  • Closed-loop zero-shot: paired task success (cluster bootstrap), step counts, wall time on benchmarks not used in training.
  • Benchmarks: held-out desktop groups, OSWorld-Verified (desktop), MobileWorld cross-application mobile suite (117 tasks: 62 memory-critical, 55 single-app controls).

Implications for AI Economics

• Resource allocation framing: CausalCache reframes memory design as per-item fidelity allocation under a fixed capacity. This is directly analogous to scarce resource allocation problems (bandwidth, attention, cache slots) — operators can treat image slots as budgeted commodities whose marginal utility should be priced or prioritized dynamically.
• Marginal utility measurement enables cost–benefit analysis: the approach provides a practical, deployable way to estimate the conditional marginal value of restoring visual fidelity for each historical event w.r.t. a fixed policy. That supports decisions like how much compute, latency, or memory budget to invest per user session and where to invest to maximize task success (ROI).
• Productization and monetization:
  • Differentiated agent tiers: fidelity allocation selectors could underlie product features (e.g., premium agents that restore more distant evidence where it matters).
  • Cost savings: holding total visual budget fixed while reallocating can improve performance without increasing storage, bandwidth, or prompt size, lowering marginal deployment cost compared to simply increasing capacity.
• Operational and market impacts:
  • Cross-platform transfer (desktop training → mobile gains) implies cheaper training regimes: selective fidelity policies can be learned in cheaper settings and transfer to resource-constrained platforms, reducing multi-platform training costs.
  • Enables finer SLA design: operators can trade latency vs. fidelity by gating second-pass re-rendering only when a high-marginal candidate displaces recent items.
• Evaluation and pricing models:
  • Benchmarks and contracts should account for fidelity allocation policies, not just static context size. Pricing of API calls / compute could reflect conditional second-pass probabilities (expected extra compute).
  • Marketplace metrics: marginal-success uplift on memory-critical tasks (e.g., +8.6 points here) provides a quantifiable lever for product value estimation and A/B decisions.
• Limitations relevant to economic deployment:
  • Gains are concentrated in memory-critical tasks; indiscriminate application can yield no benefit and could cost extra inference (second pass), so deployment must be targeted to use cases with measurable uplift.
  • Training was performed on one base policy and desktop data; selector/policy interactions may differ for other architectures or domains, so transferability and reproducibility risk must be factored into cost estimates.
  • The learned selector introduces conditional extra compute when replacements occur — operators must balance success gains against latency and GPU-cost exposure.

Overall, CausalCache offers a practical, budget-preserving way to increase agent utility by reallocating fidelity where it yields marginal value. For product managers and economists, it supplies a measurable unit of memory-value (per-slot conditional uplift), enabling principled trade-offs between compute, latency, and success-rate improvements across deployments.