← Papers

AI housing recommenders follow explicit constraints but do not optimize: 39% of suggested apartments are strictly dominated by another listing the model was shown, often costing renters hundreds of dollars a month and several minutes of commute; this failure persists across vendors and price tiers, while identity-based steering is undetectable when the candidate inventory is held constant.

Following the Preference, Missing the Optimum: Compliance Without Optimization in AI Housing Recommendation

Hsuan Lo · September 09, 2026

arxiv quasi_experimental high evidence 9/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: September 11, 2026

Hsuan Lo unresolved corpus identity

In a controlled audit using enumerated candidate pools, LLM recommenders nearly always respect stated hard constraints but frequently fail to optimize — 39% of recommendations are strictly dominated by another listing the model saw, producing substantial dollar- and time-denominated opportunity costs while showing no detectable identity-conditioned steering in the fixed-pool setting.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed September 11, 2026

View corpus context

Large language models are becoming the first point of contact for consumer search in domains where the stakes are material and the law is explicit. Existing audits show that models steer housing seekers by perceived identity, but none can say what a user loses when a recommender overlooks a suitable option, for want of an enumerated inventory to score omissions against. We audit AI housing recommendation against a verifiable ground truth. For each of 150 synthetic renter scenarios in New York City we build a pool of 120 real listings with known rent, bedrooms and GTFS-computed transit commute, compute the exact set satisfying the renter's stated constraints, and derive its Pareto frontier. The primary outcome assumes no utility function: a recommendation is strictly dominated if the same pool holds a listing cheaper, faster to commute from and no smaller in bedrooms. Across 9,945 calls to three models from two vendors, compliance is near-perfect (1.8% violation against a 66.6% random floor), yet 39.0% of recommendations are strictly dominated, and the dominating listing is a median 900 USD/month cheaper and 3.5 minutes closer. A within-scenario manipulation separates two capabilities usually conflated: changing one sentence moves median recommended rent by 646 USD/month in the correct direction, so preferences are honored, yet recommendations still sit 606 USD/month above the five cheapest qualifying listings on the same screen, and an unambiguous lexicographic instruction gives no improvement under equivalence testing against a pre-specified 50 USD/month bound. The gap widens with candidate-set size and replicates across OpenAI and Anthropic models to within 3 USD. We characterize the failure as compliance without optimization, propose dominance-rate instrumentation as a deployable diagnostic, and release all code, prompts and per-call results.

Summary

Main Finding

LLM recommenders in a controlled housing-recommendation audit “comply” with stated constraints but fail to optimize across a known candidate set. When evaluated against an enumerated inventory (150 synthetic NYC renter scenarios, 120 real listings per scenario), 39.0% of model-recommended listings were strictly dominated by another listing shown to the model (cheaper, no fewer bedrooms, shorter commute). Constraint violations were rare (1.8% overall; 0.08% over‑budget). The median dominating listing was $900/month cheaper and 3.5 minutes closer. Models respond to stated preferences but do not reliably execute the implied optimization; this failure replicates across three models and two vendors and produces a measurable welfare loss (mean rent gap vs oracle +$498/month).

Key Points

Scope and scale
- 150 synthetic renter scenarios in New York City.
- For each scenario a fixed candidate pool of 120 real listings (rent, bedrooms, GTFS-computed transit commute).
- 9,945 model API calls (9,601 parsed, 96.5%) across three models (gpt-5.6-luna, gpt-5.6-sol, claude-opus-5) and two vendors. Reported API cost ≈ US$57.
Primary outcome: strict-dominance rate
- A recommended listing is “strictly dominated” if another listing in the candidate pool is strictly cheaper, has a shorter commute, and no fewer bedrooms.
- 39.0% of recommendations were strictly dominated.
- Median dominating listing: −$900/month rent, −3.5 minutes commute.
- Mean rent deviation from oracle: +$498/month. Uniform random selection gave +$261/month, so model underperformed random on price specifically.
Constraint fidelity and mitigation
- Hard-constraint violations were uncommon: 1.8% overall; over-budget violations 0.08%.
- Deterministic pre-filtering (constraint-first) removes violations by construction.
Preference-following vs optimization
- Within-scenario manipulation: changing a single sentence to prioritize rent moved median recommended rent by $646/month and commute by 12.3 minutes in the intended direction (p < 0.0001), showing high responsiveness.
- Despite responsiveness, recommendations remained substantially above the cheapest feasible options: under “rent matters most” recommendations were +$606/month above the five cheapest feasible listings on the same screen.
- An explicit lexicographic instruction produced no material improvement; equivalence testing (TOST) established parity within a pre-specified $50/month bound (TOST p < 0.0001). Author labels the phenomenon “compliance without optimization.”
Candidate-set effects and scaling
- The optimization gap scales with candidate-set size: with filtering removed, the share of responses containing the single cheapest feasible listing fell from 93.7% at 10 candidates to 53.5% at 80, and the rent gap rose from +$111 to +$419, plateauing above ~40 candidates.
Cross-model replication and cost
- Three models spanning a ~45× token-price range produced nearly identical rent-first gaps (+$700, +$699, +$702; agreement within $3). This suggests a shared failure mode rather than a single model idiosyncrasy.
Identity-conditioned disparity
- Holding the request and candidate pool fixed, almost no identity-conditioned disparities were detected: 47 of 48 pre-specified contrasts returned null after Benjamini–Hochberg correction; 105 tract-level contrasts also returned null after correction.
- One small exception (voucher disclosure lowering Claude’s rent gap by $27.60) favored users and was below replicate-noise thresholds; overall the study reports no identity-conditioned effect in this fixed-pool ranking setting.
- Note: this null result applies to ranking over a fixed candidate pool and does not contradict prior work on open-ended steering.
Additional observations
- Information-withholding and refusal rates were 0% in all conditions.
- Dominance-rate instrumentation is proposed as a deployable diagnostic for production systems.

Data & Methods

Inventory and scenarios
- Synthetic sample: 150 renter scenarios (NYC), each paired with an enumerated candidate pool of 120 real listings.
- Verifiable attributes: monthly rent, bedroom count, transit commute computed from GTFS routing.
- Exact feasibility set and Pareto frontier computed per scenario.
Models and calls
- Models: gpt-5.6-luna (budget), gpt-5.6-sol (flagship), claude-opus-5 (Anthropic flagship).
- Temperature set to provider default; three replicates per cell to capture sampling variance.
- Total calls: 9,945 (9,601 parsed).
Primary measures
- Constraint compliance (hard-constraint violations: budget, bedrooms, commute).
- Strict-dominance indicator per recommended listing (dominated vs not).
- Monetary gap measures: mean and median rent gap versus oracle and versus random-selection baseline.
- Commute-time gap in minutes.
- Presence of cheapest feasible listing among recommendations; scaling with candidate-set size.
- Identity-conditioned contrasts (48 pre-specified) and census-tract-level contrasts (105).
Statistical procedures
- Pre-registered analysis and sampling grid.
- Benjamini–Hochberg correction for multiple comparisons in identity tests.
- Equivalence testing (TOST) with a pre-specified $50/month bound to test absence of meaningful precision gains from explicit lexicographic prompts.
- Robustness arms: varying pool density, infeasible share, candidate-set size; deterministic pre-filtering as mitigation arm.
Reproducibility
- Code, prompts, and per-call results released by the author.
- Audit focuses on API-level ranking (fixed candidate pool), not on deployed product pipelines.

Implications for AI Economics

Measurable welfare losses from ranking-stage failures
- The audit converts omission into price: the median dominating listing being $900/month cheaper (mean gap +$498/month vs oracle) implies large consumer welfare losses if left unaddressed. Even modest prevalence scales to substantial aggregate rent transfers if assistants are widely used.
Engagement metrics can be misleading
- Industry metrics (clicks, scheduled visits) may improve under LLM re-rankers while actual consumer surplus declines if cheaper/better feasible options are omitted. Economic evaluations of recommender systems should include opportunity-cost diagnostics (e.g., dominance-rate, oracle gaps) not just engagement lifts.
Design and product implications
- The separation between preference compliance and optimization suggests that LLM-based rankers may need explicit optimization modules (constraint-first filters, deterministic re-ranking, or conventional ranking algorithms) rather than sole reliance on language-model outputs.
- Deterministic pre-filtering enforces constraints but does not automatically resolve dominance/optimization; hybrid architectures should be considered (retrieval+deterministic filter + optimization/ranker).
Regulatory and fairness considerations
- The audit shows near-zero identity-conditioned disparity when ranking over a fixed candidate pool, emphasizing the importance of distinguishing retrieval vs ranking in enforcement and research. Steering observed in open-ended settings may arise upstream (retrieval) rather than in the ranker.
- Dominance-rate instrumentation provides an auditable, priced metric that can be used in regulatory contexts (e.g., fair housing enforcement) to quantify omissions and opportunity costs in natural units (dollars, minutes).
Market structure and competition
- The replication of the failure across models and price tiers implies a systemic shortcoming of current LLM ranking practices. As platforms embed LLM rankers, this could create concentrated sources of suboptimal recommendations across marketplaces, affecting price competition and consumer search efficiency.
Research and evaluation recommendations
- Recommender evaluation should include enumerated-candidate audits where feasible, and report both dominance rates and monetary/time gaps to capture welfare implications.
- Equivalence testing and pre-specified effect bounds (as used here) are useful to show the absence of meaningful improvement from prompt interventions.
- Future work should test deployed pipelines (retrieval + ranking) to locate where steering or omission occurs, and to quantify population-level welfare impacts.

Short takeaway: LLM recommenders typically follow stated preferences but often do not perform the implied constrained optimization over a known candidate set. This produces quantifiable, economically meaningful opportunity costs for renters and suggests that evaluations and deployments must move beyond surface compliance and engagement metrics to instrument and correct optimization failures.

Assessment

Paper Typequasi_experimental Evidence Strengthhigh — Strong internal validity from an enumerated ground truth and fixed candidate pools, large number of model calls (9,945, 96.5% parsed), pre-specified hypotheses and analyses (pre-registration), replicate draws per cell, multiple model vendors, and clear, directly measurable outcomes in natural units (dollars and minutes). Limitations that temper external validity are: single-city (NYC) scope, API snapshots (not full deployed pipelines), synthetic scenarios rather than live users, and inability to isolate internal model mechanisms. Methods Rigorhigh — Design implements best-practice audit features: enumerated candidate universe, verifiable objective constraints, paired within-scenario contrasts that isolate single prompt changes, pre-registration, multiple replicates, cross-model replication, formal equivalence testing, correction for multiple hypotheses, and public release of code and per-call data; weaknesses are acknowledged (no deployed-product probe, single-city sample, and limited mechanism identification). Sample150 synthetic renter scenarios in New York City; for each scenario the author assembled a candidate pool of 120 real listings with recorded rent and bedroom count and GTFS-computed transit commute times; 9,945 model calls (9,601 parsed) across three models (gpt-5.6-luna, gpt-5.6-sol, claude-opus-5) and two vendors, with a matched-identity grid of 6,840 attempted calls (6,631 parsed); three replicates per cell; pre-filter and pool-density arms also executed. Themesadoption governance IdentificationControlled API audit with enumerated candidate pools and within-scenario manipulations: for each of 150 synthetic renter scenarios the author constructs a fixed pool of 120 real listings (rent, bedrooms, GTFS-derived commute) and pins that pool while varying prompt conditions (identity cues, stated priorities, candidate-set size, pre-filtering). Causal contrasts come from paired within-scenario comparisons (holding request and inventory constant while changing only one prompt element), pre-specified hypotheses, replicate draws per cell, and multiple models/vendors to test generality; statistical control uses equivalence testing and Benjamini–Hochberg correction for multiple contrasts. GeneralizabilitySingle-city study (New York City) — results may not hold in other metropolitan or rural contexts., API-level ranking probes a pinned snapshot of models and fixed candidate pools, not full deployed products (retrieval/re-ranking pipelines vary in production)., Synthetic renter scenarios and constructed candidate pools differ from live user behavior and dynamic inventory changes., Limited to three models/vendors and the time window of calls — future model updates may change outcomes., Missing listing attributes (floor area missing for ~39% of records) and measurement of commute relies on GTFS routing assumptions.

Claims (15)

Claim	Direction	Outcome	Confidence & Evidence	Details
Models violated a renter's stated hard constraint on 1.8% of recommendations, compared with a 66.6% random-selection floor. Error Rate	positive	Rate of recommendations violating explicit budget, bedroom, or commute constraints	Reading fidelity high Study strength high	n=9945 1.8% of recommendations versus 66.6% random-selection floor 0.8
Constraint-first filtering eliminated hard-constraint violations entirely by construction. Error Rate	positive	Confirmed violations of stated hard constraints	Reading fidelity high Study strength high	eliminates violations entirely 0.8
39.0% of recommendations were strictly dominated by another listing in the same candidate pool. Output Quality	negative	Share of recommendations strictly dominated by an available alternative	Reading fidelity high Study strength high	n=150 39.0% of recommendations 0.8
When a recommendation was dominated, the dominating listing was a median $900 per month cheaper and 3.5 minutes closer in commute. Output Quality	negative	Rent and commute opportunity loss relative to a dominating listing	Reading fidelity high Study strength high	n=12281 $900/month cheaper and 3.5 minutes faster 0.8
Models had a larger rent gap relative to the cheapest qualifying option than uniform random selection: +$498 per month for the models versus +$261 for random selection. Output Quality	negative	Rent gap between selected recommendation and the cheapest feasible listing	Reading fidelity high Study strength high	+$498/month versus +$261/month 0.8
Changing one stated preference moved the median recommended rent by $646 per month in the intended direction. Task Allocation	positive	Change in median recommended rent following a stated preference change	Reading fidelity high Study strength high	$646/month 0.8
Despite responding to the stated rent priority, recommendations remained $606 per month above the five cheapest qualifying listings in the same pool. Output Quality	negative	Residual rent gap after a rent-first instruction	Reading fidelity high Study strength high	+$606/month 0.8
An unambiguous lexicographic instruction did not materially improve optimization: the full-grid effect was +$3.48 per month, with a 90% confidence interval of [-$11, +$18], within the prespecified $50 equivalence bound. Output Quality	null_result	Change in rent gap from an unambiguous lexicographic instruction	Reading fidelity high Study strength high	+$3.48/month; 90% CI [-$11, +$18] 0.8
As the candidate set grew from 10 to 80 listings, the share of responses containing the single cheapest available listing fell from 93.7% to 53.5%. Output Quality	negative	Probability that the recommendation contains the cheapest available listing	Reading fidelity high Study strength medium	93.7% at 10 candidates versus 53.5% at 80 candidates 0.48
The rent gap increased from +$111 at 10 candidates to +$419 at 80 candidates and plateaued above 40 candidates. Output Quality	negative	Rent gap relative to the cheapest available listing as candidate-set size increases	Reading fidelity high Study strength medium	+$111/month at 10 candidates to +$419/month at 80 candidates 0.48
The rent-first gap was nearly identical across the three audited models: +$700, +$699, and +$702 per month. Output Quality	mixed	Rent gap under the rent-first condition across models	Reading fidelity high Study strength medium	n=60 +$700, +$699, and +$702 per month 0.48
Claude had a lower general dominance rate than the OpenAI models: 34.3% versus 39.4%. Output Quality	positive	Rate of recommendations strictly dominated by another listing	Reading fidelity high Study strength medium	n=60 34.3% versus 39.4% 0.48
After Benjamini-Hochberg correction, 47 of 48 prespecified identity-conditioned contrasts were null. Inequality	null_result	Differences in rent gap, commute gap, dominance rate, or neighborhood exposure across identity conditions	Reading fidelity high Study strength medium	n=6840 47 of 48 contrasts null after correction 0.48
An additional 105 contrasts involving census-tract characteristics of recommended listings found no significant effects after correction. Inequality	null_result	Identity-conditioned differences in recommended-neighborhood census-tract characteristics	Reading fidelity high Study strength medium	n=105 No significant effect after correction 0.48
Refusal and information-withholding rates were 0.0% in all identity conditions. Ai Safety And Ethics	null_result	Rates of refusal and withholding of housing information	Reading fidelity high Study strength medium	n=6840 0.0% in all conditions 0.48