AI assistants favor highly rated and higher-priced hotels and even a search list position — a content-free artifact — meaningfully shifts recommendations worth roughly $12 per night; eco-labels get outsized weight while manager replies are largely disregarded.

Main Finding

LLM travel assistants select hotels largely the way humans do on the headline margins—guest rating and price dominate recommendations—but they reweight some reputation cues and introduce a measurable positional distortion. Concretely (pooled across 12 models): a top guest rating raises recommendation probability by 31.6 percentage points, a high price lowers it by 30.0 pp, eco-certification adds +11.6 pp, review volume +8.3 pp, chain affiliation −1.8 pp (small penalty), and visible management response has no detectable effect (+0.1 pp). Independently randomized list position causally shifts recommendations; the first slot is worth about $12/night on average (with substantial model heterogeneity). Models’ stated reasons correlate with but imperfectly reflect the causal weights (Spearman ρ ≈ 0.59–0.85). Results are robust across personas, prompt paraphrases, decoding temperatures, and layout/order variants.

Key Points

• Primary drivers
  • Guest rating (valence) and price are the dominant causal determinants of an LLM’s hotel recommendation, mirroring human eWOM ordering on these two margins.
  • Magnitudes (pooled AMCEs): rating (top) +31.6 pp; high price −30.0 pp.
• Secondary and muted signals
  • Eco-certification is materially rewarded by LLMs (+11.6 pp), more than typical human benchmarks suggest.
  • Review volume gives a positive but smaller boost (+8.3 pp).
  • Management response—commonly promoted in optimization advice—shows no detectable effect.
  • Brand/chain affiliation has a negligible-to-slightly-negative effect overall (−1.8 pp), and its effect varies by persona.
• Position bias
  • List position, a content-free artifact, causally changes recommendations; pooled first-slot advantage ≈ $12/night (≈ a tenth of a rating step).
  • Position effects vary widely across models—some show much larger order sensitivity.
• Persona heterogeneity
  • Price sensitivity is highest for the budget-family persona.
  • Eco-certification is most rewarded for the eco-conscious persona.
  • Chain effects vary by persona: the pooled chain penalty is concentrated in some personas and can vanish for business travelers.
• Stated vs revealed reasons
  • Models’ textual reasons partially track revealed causal weights (positive correlations), but they over-mention brand and underreport positional/list-order influences.
• Robustness and pre-specification
  • Design, estimands and hypotheses (H1–H12) were pre-registered/hashed. Findings hold across nine prompt paraphrases, two decoding temperatures (0.0, 0.7), card/snippet layouts, and field orders.

Data & Methods

• Audit design
  • Pre-specified, randomized choice-based conjoint (algorithm audit) presenting an assistant with five synthetic hotel cards per choice set.
  • Independently randomized attributes per card: guest rating, review volume, review recency, management response (visible or not), chain affiliation (brand vs independent), nightly price (low/medium/high), Green Key eco-certification, plus list position randomized independently.
  • Each model asked to (a) pick one recommended hotel from the five and (b) state a reason.
• Estimand and analysis
  • Primary estimand: Average Marginal Component Effect (AMCE) of each attribute on P(recommendation), reported in percentage points.
  • Secondary specs: conditional logit models, price-equivalent and rating-step-equivalent conversions, interaction tests (e.g., rating×volume, volume×recency), multiple-comparison corrections, equivalence tests for null effects.
• Experimental scope and pre-registration
  • Focus limited to the selection (gatekeeping) stage among a fixed candidate set (not retrieval or multi-turn negotiation).
  • Confirmatory hypotheses H1–H12 specified in advance; full code, prompts, and outputs archived.
• Models, sample size, and variations
  • Panel of 12 models: 4 open-weight LLMs run locally (Llama-3.2-3B, Qwen-2.5-3B, Phi-3-mini, Mixtral-8×7B) and 8 proprietary API-access models (OpenAI GPT-4o-mini; Google Gemini 1.5 Pro, 2.0 Flash, 2.0 Pro; four Anthropic Claude variants).
  • Three traveler personas, nine prompt paraphrases, two decoding temperatures.
  • 3,024 main-arm choice sets per model; >60,000 model calls total.
• Key robustness checks
  • Persona-conditioned interactions, model fixed effects, prompt paraphrase and temperature robustness, layouts and field order checks.
  • Stated-versus-revealed comparison via Spearman rank correlations between attributes named in rationales and AMCE magnitudes.

Implications for AI Economics

• Market visibility and distributional effects
  • LLMs acting as gatekeepers can create concentrated, asymmetric visibility effects: small differences in rating, price, eco-cert status, or list position translate into large swings in recommendation probability.
  • Position bias (priced at ≈ $12/night pooled) implies platform/UI design choices and internal ranking heuristics can shift revenues substantially across suppliers; this amplifies the economic importance of platform placement even within conversational interfaces.
• Strategic response and “generative engine optimization”
  • Hotels and intermediaries will rationally adapt (and already are adapting) to these causal weights. The evidence suggests prioritizing actions that increase observable rating and manage price signals; eco-certification investments may yield higher returns in LLM-mediated channels than expected from human-only benchmarks.
  • Optimization advice should be grounded in measured causal weights: the audit shows some industry-recommended levers (e.g., management response) may be ineffective for LLM recommendation.
• Platform accountability, transparency, and regulation
  • Models’ stated reasons are only an imperfect proxy for what drives outcomes; “explainability” claims that rely on textual rationales risk misrepresenting actual decision drivers.
  • Quantified position-equivalents give regulators and platform designers concrete metrics to assess and mitigate order-induced distortions (e.g., counterfactual re-ranking, mandatory disclosure of selection criteria, randomized auditing).
• Welfare and competitive concerns
  • Reweighting of eco-certification and relative neglect of management response/brand could reallocate demand across hotel types in ways not aligned with consumer welfare or producer expectations.
  • The gatekeeping power of LLMs makes entry and survival dynamics more sensitive to platform-mediated signals, potentially increasing incentives for gaming, manipulation of visible metadata, or paid placement-like arrangements within chat interfaces.
• Research and policy agenda for AI economics
  • Need for more causal audits across additional stages (retrieval, multi-turn negotiation, booking conversion) and more diverse models/platforms to map end-to-end market effects.
  • Standardize price-equivalent and rating-equivalent metrics for cross-study comparability so economists and managers can translate algorithmic biases into economic magnitudes.
  • Consider regulatory standards for verifiable auditing, required disclosure of selection logic, and limits on positional advantage in conversational commerce.
• Practical takeaways for stakeholders
  • Hotels: invest first in improving average guest rating and understand price-position trade-offs; eco-certification may be a higher-return lever in LLM channels than in traditional channels.
  • Platforms: mitigate position bias, surface transparent selection criteria, and monitor model heterogeneity to avoid large, model-specific allocation shocks.
  • Regulators and economists: treat LLMs as consequential, opaque intermediaries; prioritize causal audits to inform policy and competitive assessments.

Limitations to keep in mind: the audit isolates the selection among fixed candidates (not retrieval), uses synthetic personas and design choices fixed in time, and evaluates a snapshot of models available during the study period—effects may evolve with model updates, platform integration, or different prompt/UX conventions.