LLM advice speeds decision-making in simulated search-and-rescue but not overall rescue outcomes: assistance raises victims-per-step and rewards as users attend more to chat, yet total victims saved remains unchanged because novices over-rely on AI while experts cross-check guidance to preserve situational awareness.

Main Finding

LLM guidance in a simulated search-and-rescue (SAR) task improved operational efficiency (higher victims-per-step and higher total rewards) but did not increase absolute task throughput (number of victims saved). Eye-tracking shows attention shifted from the environment to the chat interface and cognitive-load markers increased. Expertise moderated these effects: novices tended toward passive reliance on the LLM, while experts preserved an active verification loop (more environment scanning and attenuated pupil variability), suggesting LLMs act as a skill-biased aid that changes how operators allocate attention rather than simply increasing output.

Key Points

• Performance
  • LLM vs No-LLM: significant increases in victims-per-step (β = 0.02, q < 0.01) and total rewards (β = 267.14, q < 0.01).
  • No significant effect on total number of victims rescued (β = 0.035, q = 0.8973).
• Attention and processing (eye-tracking)
  • Mean fixation duration decreased under LLM guidance (β = −19.33 ms, q < 0.01).
  • Fixation time shifted away from the game area and toward the chat interface (game area fixation time β = −0.184, chat area β = +0.092; both q < 0.0001).
  • Pupil diameter variability increased with LLM use (β = +0.122, q < 0.0001) — interpreted as increased cognitive load/arousal.
• Expertise moderation
  • LLM × Expertise interactions indicate experts attenuated some cognitive costs of LLM use: reduced pupil-STD increase (interaction β = −0.051, q < 0.05) and relatively greater fixation time retained in the game area under LLM (interaction β = +0.114, q < 0.01).
  • Experts showed increases in number of fixations and saccades under LLM (LLM×Expertise βs ≈ +0.8–0.96, q < 0.05–0.01), consistent with active cross-checking; novices showed more passive reliance.
• Interpretation
  • LLMs function as informational teammates that offload some information acquisition (shorter fixations, more attention to chat) and improve decision efficiency, but do not necessarily raise total saved victims within time limits.
  • Effective human-AI teaming depends on the human operator’s ability to verify AI guidance against environment cues — a behavior more common in experts.

Data & Methods

• Task and environment
  • Grid-based, partially observable SAR simulation (MiniGrid-based) with victims, hazards, doors/keys; victims deteriorate over time with hidden health values.
  • Participants rescue victims by moving adjacent and issuing a pickup action.
  • Victim health values were hidden by default and revealed for 5 seconds only when participants requested LLM guidance (ALT).
  • Scoring: +10 for rescuing a healthy victim, −10 for picking a decoy, −20 for rescuing a dead victim (asymmetric penalties to encourage cautious prioritization).
• LLM assistance
  • LLM acted as a “radio coordinator” giving concise directional instructions (N/S/E/W), choosing a single immediate objective via a fixed decision hierarchy (savable nearest victims → best remaining → keys/doors → explore).
  • State encoded with a BFS-based structured table input (performed best vs ASCII or semantic formats).
  • Two LLM-guided conditions and one No-LLM baseline per participant; responses constrained to short action-oriented messages.
• Participants & individual differences
  • N = 13 university participants (normal vision, basic computer familiarity).
  • Each completed three randomized 15-minute runs (two LLM runs, one baseline).
  • Pre-task assessments: Multiple Object Tracking (MOT) and Visual Search (VS); K-means clustering (k=2) with MOT & VS metrics produced Expert vs Novice groups.
• Measures
  • Performance: victims-per-step (efficiency), total rewards, number of saved victims.
  • Eye-tracking: Tobii at ~60 Hz; metrics included fixation count/duration, AOI fixation times (game vs chat), saccade counts/amplitudes, and pupil diameter SD.
• Analysis
  • Run-level linear mixed-effects models with participant random intercepts: predictors LLM condition, Expertise, LLM×Expertise; count outcomes log-transformed as needed.
  • Benjamini–Hochberg FDR correction applied to p-values.
• Limitations (reported/implied)
  • Small sample (N=13), lab-based simulated task, short exposure per condition (15-minute runs), health-reveal mechanic (information asymmetry) specific to design.

Implications for AI Economics

• Productivity vs Efficiency: LLMs can raise per-action efficiency (reduce wasted movements/actions) and improve reward-weighted decision quality without necessarily increasing absolute output under time constraints. Economic models of AI-driven productivity should distinguish efficiency gains (time/effort per task unit) from throughput or total output.
• Skill-biased complementarity and substitution
  • Complementarity for experts: experts maintained verification behaviors and bore less cognitive cost, implying LLMs can augment high-skill workers’ performance and decision quality.
  • Potential substitution or deskilling for novices: novices were more likely to accept LLM outputs passively, which may reduce skill acquisition and increase automation complacency. Labor-market effects could include reallocation of tasks (experts handle oversight/verification; novices handle lower-risk or LLM-supervised tasks) and changes in skill premium.
• Human capital and training investment
  • Because effective use requires cross-checking and situational awareness, firms may need to invest in training that preserves verification skills and teaches when/how to override AI outputs. Failure to do so risks elevated cognitive load or safety risks in high-stakes tasks.
• Monitoring, incentives, and design of work systems
  • Productivity metrics and incentives should capture both efficiency and safety-related behaviors (e.g., verification effort), not just raw throughput. Payment schemes that reward only rescue counts could unintentionally encourage risky behavior if LLMs encourage faster but not safer actions.
  • Interface design matters: LLMs that shift attention to a chat pane can create an attention-guidance trade-off. Economically, this implies potential hidden costs (increased monitoring or error rates) unless interfaces are designed to support quick cross-referencing or highlight actionable uncertainties.
• Regulation and risk management
  • In regulated or safety-critical sectors, oversight protocols and mandatory verification steps may be necessary when deploying LLM guidance to mitigate automation bias and cognitive overload.
• Measurement and evaluation
  • Standard output metrics understate the full economic impact of LLM assistance. Firms and policymakers should incorporate behavioral and cognitive measures (attention, pupil-based load proxies, verification time) into cost–benefit analyses of AI deployment.
• Research and deployment recommendations
  • Scale evidence: small lab sample suggests patterns but larger/naturalistic deployments needed to estimate aggregate labor-market effects and returns to AI by skill level.
  • Design for mixed-ability teams: allocate tasks to exploit complementarity (experts as verifiers, LLMs as information aggregators), and create feedback loops that sustain operators’ situational-awareness skills.
  • Consider adaptive interfaces or constrained automation that preserves required human monitoring to avoid efficiency gains that increase downstream costs (errors, rescues of dead victims, liability).

Overall, the study highlights that LLMs can improve task efficiency but reshape attention and cognitive workload in ways that depend on operator expertise. From an AI-economics perspective, this implies heterogeneous returns to AI across worker skill levels, nontrivial training and monitoring costs, and the need to evaluate AI benefits beyond simple throughput measures.