Large language models reveal measurable willingness-to-pay in travel choice tests but tend to overvalue options relative to humans, especially for expensive choices and business personas; telling models about prior cheap preferences reduces this bias.

Main Finding

Large language models (LLMs) can be coerced into revealing economically interpretable preferences in the form of willingness-to-pay (WTP) estimates when placed in multi-attribute, subjective choice tasks (hotel-room selection). For sufficiently large models (Llama 3.3 70B, GPT-4o, Gemini-3.1-Pro) the authors obtain meaningful multinomial-logit fits and WTPs, but the derived values systematically deviate from human benchmark estimates (Masiero et al., 2015). Crucially, these WTPs are model- and prompt-dependent: providing persona information or expensive historical examples pushes WTPs up, while cheap examples or a budget/student persona pull WTPs down. Smaller LLMs produced unreliable behavior (order bias, positive price coefficient, low explanatory power) and were excluded.

Key Points

• Experimental setup: 240 binary choice dilemmas built from all combinatorial hotel-room alternatives (based on Masiero et al., 2015). Attributes: view (city/harbour), floor (10/18/26), club access, minibar level, guest smartphone, cancellation (non-refundable/refundable), price (HK$1600–3200).
• Prompting conditions (user info) tested:
  • No user info (baseline)
  • In-context learning (ICL) examples (various configurations)
  • Persona prompting (business or student)
  • Combination of ICL + persona
• LLMs tested (main results): Llama 3.3 70B, GPT-4o, Gemini-3.1-Pro. Temperature set to 0 for main experiments; smaller models (multiple 3B–8B checkpoints) yielded poor/illogical econometric fits and were dropped.
• Estimation: for each (model, prompt) the LLM’s choices across 240 dilemmas are treated as observations; a multinomial logit model is fitted on standardized attributes. WTP for attribute k is derived from the estimated coefficients via the usual attribute/price coefficient ratio (implemented as WTPk = βk * σ_price / (βprice * σk) in the paper).
• Findings on WTPs:
  • Larger LLMs produce interpretable utility parameters and WTPs that can be rationalized by a discrete-choice model.
  • Compared to human reference WTPs, LLM-derived WTPs are generally higher on average, but attribute-level deviations exist (direction and magnitude vary by model and prompt).
  • Context (ICL, persona) systematically shifts WTPs: business persona and expensive-example histories → upward shift; student persona and cheap-example histories → downward shift.
• Robustness checks: WTPs are sensitive (but not entirely brittle) to prompt paraphrasing, prompt length, option order switching, currency denomination, and sampling temperature.
• Reproducibility: authors release their codebase and experimental setup.

Data & Methods

• Choice-set generation:
  • Origin: attributes and levels from Masiero et al. (2015) for a medium-sized Hong Kong hotel.
  • All 480 possible unique alternatives enumerated; 240 random pairwise dilemmas constructed such that each option appears only once.
• Prompt template:
  • A deterministic instruction telling the model it must choose A or B for the user (only output A or B).
  • Optional inserted user information: ICL examples, persona text, or both.
  • Alternate descriptions follow a fixed natural-language format listing the seven attributes and price.
• LLMs and configuration:
  • Primary models: meta-llama/Llama-3.3-70B-Instruct, gpt-4o-2024-08-06, gemini-3.1-pro-preview.
  • Main experiments run at temperature = 0; additional temperature checks conducted.
  • Smaller LLMs (Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 7B, Haiku) produced unreliable econometric output and were not reported.
• Econometric modeling:
  • For each (model, prompt) combination the simulated binary choices y(i) ∈ {A,B} over i = 1..240 are recorded.
  • Multinomial logit is fit (xlogit package, BFGS optimizer) on standardized attributes; no train/test split because goal is explanation, not prediction.
  • WTP computation: attribute coefficient scaled relative to price coefficient and feature standard deviations to obtain interpretable monetary trade-offs.
• Robustness analyses: paraphrase prompts, change order of alternatives, change currency denominations, vary temperature, test prompt length and example composition.

Implications for AI Economics

• LLMs as preference elicitors: Large LLMs can be used to extract structured, interpretable preference signals (WTPs) in subjective choice domains. This opens a route to rapid, low-cost simulation of preference trade-offs for economic research or product-feature valuation.
• No ground-truth guarantee: LLM-derived WTPs are descriptive model outputs, not objective measures of human utility. They should not be treated as ground truth for real consumers without calibration to human data.
• Sensitivity and manipulability:
  • WTPs are sensitive to prompt framing, example selection, and persona specification. This is double-edged: it enables personalization (steer an agent toward user types) but also means small prompting changes can materially alter economic recommendations.
  • Model- and prompt-dependence implies that marketplace analytics, pricing decisions, or automated agentic choices based on raw LLM-extracted WTPs risk systematic biases unless carefully validated.
• Model selection matters:
  • Large models behave sufficiently coherently for econometric modeling; small models often fail (order bias, wrong sign on price). Economic practitioners should avoid relying on low-capacity LLMs for preference elicitation.
• Practical recommendations for deployment and research:
  • Always benchmark and calibrate LLM-derived WTPs against human experimental data before using them to make economic decisions.
  • Treat persona and history prompts as policy levers—use them intentionally to capture user heterogeneity, but audit for unwanted stereotyping or manipulation.
  • Run robustness checks (paraphrasing, order randomization, temperature sweeps) to detect brittle preferences and prompt-dependence.
  • Consider finetuning or preference-conditioning methods if alignment to a target population is required.
• Research opportunities:
  • Extend beyond hotel contexts to other consumer domains to test external validity.
  • Investigate methods to debias or calibrate LLM WTPs to human distributions (e.g., RLHF anchored to human choice data).
  • Explore richer econometric models (hierarchical or mixed logit) when modeling populations of users represented by distinct personas.

Limitations noted by the authors: the study is descriptive (LLM WTPs are compared to a human benchmark for interpretation, not to establish truth), results are domain-specific (hotel choice in HK), and smaller models often cannot be meaningfully used for WTP extraction. The full code and experimental material are made publicly available for replication.