AI shopping agents still show position bias, but it is weaker and often 'lost-in-the-middle'; crucially, inspection biases rarely change bookings, which overwhelmingly gravitate to a single undominated hotel, implying SEO placement matters less when consumers delegate purchases to AI.

Main Finding

AI agents (LLMs) search more deeply than humans and still exhibit position bias when inspecting listings, but that bias is much weaker and non-monotonic (a "lost‑in‑the‑middle" U‑shape). Crucially, the residual position effects on inspection do not reliably translate into different choices: agents overwhelmingly converge on a single undominated listing and always book (100% conversion), meaning placement matters less than the attributes shown on the results page. Increasing an agent’s reasoning effort further attenuates positional bias.

Key Points

• Depth of search:
  • AI agents inspect substantially more listings per session than humans: means = 1.63 (Claude Sonnet 5), 3.12 (Gemini 3.1 Flash Lite), 4.25 (Gemini 3.7 Flash), 5.83 (Gemini 3.1 Pro) vs human benchmark 1.12.
  • Degenerate consideration sets (one inspection) are rare for some models (0.8%–37.4% vs 93% for humans).
• Conversion / outside option:
  • Every AI agent booked in 100% of sessions (versus 66% conversion and 34% outside option for humans).
• Position effect on inspection:
  • Position still predicts inspection causally (order randomized), but effects are 4–10× smaller than for humans. Linear position coefficients on inspection range roughly −0.0002 to −0.0005 (human ≈ −0.0019).
  • For three of four LLMs, inspection probability declines from the top to a minimum around ranks 68–74 then rises toward the bottom (lost‑in‑the‑middle U‑shape); Gemini 3.1 Pro shows mostly primacy without the recency rise.
• Position effect on choice:
  • Position rarely affects final booking. Choice position coefficients are indistinguishable from zero for two models and very small for others. Average chosen ranks ≈ 44.0–49.7 (neutral = 50.5).
  • Choices concentrate on a single undominated hotel (same modal choice across models), which accounted for 78.2% of all bookings pooled.
• Reasoning effort:
  • Manipulating reasoning effort (multiple levels across Gemini Flash Lite and Gemini Pro) reduces the lost‑in‑the‑middle curvature; at higher effort levels the positional curvature and the position effect on choice become insignificant.
• Controls and other predictors:
  • Attributes visible on the results page (review score, price, promotion, chain) strongly predict inspection and choice; review score is a major heuristic driving choices.
• Heterogeneity:
  • Model behavior differs substantially across LLMs in search depth and inspection patterns; this heterogeneity does not map cleanly to provider or capability tier alone.

Data & Methods

• Setup:
  • Randomized controlled experiment that randomized the order of 100 hotel listings per session (to eliminate ranking-quality endogeneity).
  • Two-layer information structure: listing page (name, review score, nightly price, total price, aggregated rating) and an inspect tool that returns detailed hotel attributes when called (analogue of a click).
  • Agents received the full 100-item page in the context window and could sequentially call inspect and then submit_choice (or terminate without booking).
• Models and sample:
  • Main experiment: 4 LLMs (Google Gemini 3.1 Pro, Gemini 3.7 Flash, Gemini 3.1 Flash Lite; Anthropic Claude Sonnet 5), 500 independent sessions per model → 2,000 sessions.
  • Follow-ups: reasoning effort manipulation across 7 cells (Gemini Flash Lite: 4 effort levels; Gemini Pro: 3 levels), 500 replications per cell → 3,500 sessions.
• Measures and estimation:
  • For each session×hotel: Inspected (1 if inspect tool used), Chosen (1 if booked). Position indexed 1–100 random per session.
  • Controls: price (nightly), review score (aggregated), chain (brand binary), promotion (discount badge binary).
  • Analysis: linear probability models with session-clustered standard errors; tests for nonlinear (quadratic) position terms to identify U‑shape.
• Benchmarks:
  • Results compared to human field benchmark (Ursu 2018), acknowledging differences in list length and some displayed attributes.

Implications for AI Economics

• Market value of ranking slots will change under delegated search:
  • The traditional premium for top positions (driven by limited human attention and sequential scanning) weakens when consumers delegate to agents that can process full pages and deliberate.
  • Platforms that monetize placement (paid search, sponsored slots, auctions) may see reduced willingness to pay for top slots; price discrimination strategies and auction design may need revision.
• Importance of on‑page attributes over placement:
  • For agentic delegation, visible quality signals (review scores, price, promotions) matter more for final demand than rank. Managers should reallocate SEO/marketing spend toward improving and signaling attributes rather than only chasing top placement.
• Platform and recommender design:
  • Recommender interfaces and APIs should prioritize clear, structured attribute fields that agents can reliably use (since agents rely on attributes), and consider how context length and information ordering interact with model retrieval behaviors (lost‑in‑the‑middle).
  • Providing structured metadata (high-quality ratings, canonical prices) may be more effective than position tweaks for influencing agent-mediated sales.
• Strategic behavior and competitive dynamics:
  • With agents converging on undominated options, winner-take-most dynamics from a quality signal perspective may intensify (one listing capturing large share), shifting competition toward improving objective dominance (better price/score combinations).
• Policy and consumer welfare:
  • Agents’ tendency to always book and to focus on a single modal option raises questions about user control, outside-option use, and necessity of transparency/disclosure for delegated decisions.
  • Regulators and platforms may need to consider standards for how search results are structured for agent consumption, and whether agents should be required to surface alternatives or explanations.
• Research directions:
  • Evaluate real-world outcomes where users actively choose to delegate to personal agents (endogenous delegation), other product categories, multi-stakeholder settings (advertisers/platforms), and strategic listing behavior when agents are common.
  • Study how personalization and agent objectives (e.g., risk aversion, client preferences) interact with position/attribute effects and market equilibria.

Limitations worth noting: lab-style agent experiments use default prompts and provider APIs (sampling parameters often not adjustable); external validity depends on how real users compose prompts, select agents, and constrain agent behavior; and results may vary with different listing lengths, domains, or richer attribute sets.