An XR copilot that learns from both behavior and passive EEG increases users' target throughput in a demanding search-and-act task and restores correct guidance faster after silent task changes; combining implicit neural signals with explicit actions yields the largest and most reliable within-session performance gain in the lab.

Main Finding

OLIVE is an online latent-inference framework that fuses sparse explicit behavior (shots) and fixation-locked EEG (implicit physiological evidence) to adapt a frozen vision–language model in real time. In an XR first‑person shooter surrogate for attention-limited, time‑critical tasks, OLIVE (when using both evidence channels, “OLIVE‑IE”) Pareto‑dominates single‑modality and prior test‑time adaptation baselines, converges rapidly within active task rounds (96.8% guidance convergence in 28.3 s), and produces the largest, most reliable within‑session improvement in operator throughput. When the target class shifts silently, OLIVE‑IE reconverges to correct guidance substantially faster than a behavior‑only agent (≈1.27× faster on average; 14.7 s faster in absolute terms; p = .008), restoring trustworthy assistance when it matters most.

Key Points

• Problem setting
  • Targets arrive faster than a human can inspect and must be acted on within expiring windows (operational vigilance tasks).
  • Ground truth labels are structurally incomplete: users act on only a subset of items; absence of action is not a negative label.
• OLIVE algorithm
  • Maintains item-level posteriors over latent target state z_i and jointly estimates per-source reliabilities via an online EM loop.
  • Evidence channels: visual crop scored by a frozen VLM (CLIP ViT‑B/16 with a small learnable virtual prompt), explicit behavioral labels (shots), and implicit per‑fixation probabilities from a fixation‑locked EEG decoder (FRP/P300/N2pc signal).
  • Prompt tuning (virtual prompt only) adapts the frozen VLM quickly and parameter‑efficiently at test time.
  • Decoder‑agnostic design: OLIVE treats any per‑fixation probability as an evidence source and estimates its reliability online.
• Empirical performance
  • In simulation and user studies, OLIVE with both EEG and behavior (OLIVE‑IE) outperforms single‑channel variants and baselines (TTA, crowd‑EM variants, RL) on convergence rate and reliability.
  • Key reported effects (live deployments):
  • Guidance convergence: 96.8% in 28.3 s (first third of a 90 s round).
  • Sustained workload (US2): OLIVE‑IE increased target throughput by Δ = +0.031 kills/s (p = .003), independent of baseline shooting skill.
  • Silent target shifts (US3): OLIVE‑IE improved most on new target class (Δ = +0.070 kills/s, p = .001) and reconverged 1.27× faster (14.7 s faster in absolute time) than behavior‑only agent (p = .008).
• Practical system
  • Online latency kept < 10 ms per visual frame; virtual prompts and EM updates are run within task time budget.
  • Requires a short calibration visual search phase to train a participant‑specific FRP decoder (but OLIVE can treat decoders as low‑reliability if population pretrained).
• Reproducibility
  • Code/implementation and a de‑identified dataset of per‑fixation probabilities and metadata are available (links provided in paper).

Data & Methods

• Task and participants
  • XR SpaceShooter surrogate task: defend a mothership against mixed fleet; initial target prevalence ≈ 0.30; asymmetric cost for false positives vs misses; rounds are 90 s.
  • Difficulty adapts per round (QUEST+ staircase) to maintain ≈70% round score.
  • Three user studies: (US1) convergence behavior, (US2) live deployment under sustained load, (US3) repeated silent target switches to measure reconvergence.
• Physiological pipeline
  • Fixation‑locked EEG epochs centered [−200, +800] ms around fixation onset; standard preprocessing (bandpass, baseline, ICA for eye‑artifact suppression).
  • Per‑participant FRP decoder trained offline in a Visual Search calibration phase using labeled static arrays; decoder outputs per‑fixation target probability ˆy_physio ∈ (0,1).
  • Pupillometry (TEPR) excluded in implementation due to XR luminance confounds.
• OLIVE inference
  • Item-level latent variables z_i (binary targetness) with prior π (≈0.30).
  • Evidence likelihoods: visual scorer Λ_vis from CLIP prompt similarity, explicit shot labels (positive‑only, sparse), implicit soft labels from EEG decoder.
  • EM loop jointly updates: posterior μ_i = P(z_i = 1 | evidence), per-source reliability parameters, and the virtual prompt parameters (prompt‑tuned) — all online.
• Baselines and comparisons
  • Single‑modality variants (visual only, behavior only, physio only).
  • Algorithmic baselines: test‑time adaptation (entropy minimization/prompt tuning without human evidence), crowdsourcing/EM label aggregation variants, reinforcement learning baselines.
  • Evaluation metrics: guidance convergence rate and time, within‑session throughput change (kills/s), convergence speed, robustness under cognitive overload.
• Statistical reporting
  • Convergence and throughput improvements reported with p‑values (e.g., Δ = +0.031 kills/s, p = .003; Δ = +0.070 kills/s, p = .001; reconvergence p = .008).
• Resources
  • Implementation and reproduction scripts: public repository; de‑identified dataset of per‑fixation probabilities available.

Implications for AI Economics

• Productivity augmentation and complementarities
  • OLIVE demonstrates that fusing implicit neural signals with sparse behavior can increase operator throughput in attention‑limited tasks. In economic terms, this is a skill‑augmenting technology that raises effective labor productivity on short, high‑stakes tasks where attention is the bottleneck.
  • Because gains were largely independent of baseline shooting skill, adoption could compress performance variance across workers—raising the floor more than shifting the ceiling—which has implications for hiring, training investments, and skill premia.
• Cost / deployment tradeoffs
  • Upfront and per‑user costs: wearable EEG hardware, calibration time (per‑participant decoder training), and integration with perception backends. These costs must be balanced against time‑savings and error‑reduction in the target domain.
  • Latency and compute are low (online updates < 10 ms and parameter‑efficient prompt tuning), which favors deployment in real‑time operational settings where rapid adaptation matters.
• Dynamic tasks and value of responsiveness
  • OLIVE’s faster reconvergence after silent target shifts reduces the duration of misaligned assistance; in real OV domains, faster recovery can meaningfully lower the expected cost of misses/false alarms. Quantifying these avoided costs is central to ROI calculations for procurement decisions.
• Labor demand and task allocation
  • Systems like OLIVE may change the division of labor between humans and AI: humans remain final decision makers but can handle higher candidate throughput. This could reduce the need for highly specialized or highly experienced operators in certain monitoring roles while increasing demand for roles that supervise and maintain adaptive assistance systems.
• Market and product opportunities
  • There is an emergent product market for “neural‑augmented copilots” in ATC, radiology triage, surveillance, and other OV domains. Vendors can differentiate on calibration overhead, robustness of physiological decoding, and integration with domain models.
• Privacy, regulation, and trust economics
  • Implicit neural sensing raises privacy, consent, and governance issues that affect adoption costs and regulatory risk. Economic adoption models must factor in compliance, liability, and potential trust losses if neural signals are perceived as intrusive or misused.
  • Trust calibration is directly relevant to economic value: faster, well‑calibrated reconvergence reduces the likelihood of user distrust and consequent disuse (which would negate productivity gains).
• Research and evaluation needs for economic deployment
  • Cost‑benefit studies comparing OLIVE vs cheaper alternatives (e.g., eye‑tracking alone, improved UI/alerts, additional staffing).
  • Longitudinal studies to measure durable productivity gains, learning effects, and labor market impacts.
  • Sector‑specific pilots that translate throughput gains (e.g., kills/s) into domain metrics (e.g., items triaged per hour, cases reviewed per shift, reduction in missed critical events) to inform procurement and policy decisions.

If you want, I can (1) extract the key numerical results and experimental design details into a one‑page table for rapid policy/ROI review, or (2) draft a short memo outlining how to evaluate the cost‑effectiveness of deploying OLIVE‑style systems in a specific OV domain (e.g., radiology triage or air traffic monitoring).