Large language models often treat stored user preferences as global rules rather than context‑dependent signals, leaking or applying preferences in third‑party contexts; stronger personalization improves correct tailoring but also raises harmful misapplication, and prompt‑based fixes only partially mitigate the problem.

Main Finding

BenchPreS (Benchmark for Preference Suppression) shows that modern LLMs frequently misapply persistent user preferences in contexts where social or institutional norms require suppression (third‑party communication). Using two complementary metrics, Misapplication Rate (MR) and Appropriate Application Rate (AAR), the paper demonstrates a pervasive context‑sensitivity failure: models often treat stored preferences as globally enforceable rules rather than context‑dependent normative signals. Stronger adherence to preferences increases correct personalization but also increases harmful over‑application, and neither improved chain‑of‑thought reasoning nor prompt‑based defenses fully eliminate the problem.

Key Points

• Problem framing: Many LLMs store user preferences in persistent memory to personalize across interactions, but some settings (e.g., third‑party communications, legal or workplace contexts) require suppression of those preferences for social, ethical, or institutional reasons.
• BenchPreS: A benchmark and evaluation protocol that assesses whether an LLM applies or suppresses stored user preferences appropriately across varied communication contexts.
• Metrics:
  • Misapplication Rate (MR): fraction of instances where a preference was applied even though the context required suppression.
  • Appropriate Application Rate (AAR): fraction of instances where a preference was applied when it was appropriate to do so.
• Empirical findings:
  • Frontier LLMs struggle to make context‑sensitive decisions about when to apply preferences.
  • Models that more faithfully enforce stored preferences achieve higher AAR but also systematically have higher MR (trade‑off between personalization and over‑application).
  • Attempts to fix the behavior with stronger reasoning prompts (e.g., chain‑of‑thought) or with prompt‑based safety/defenses reduce but do not eliminate misapplication.
• Interpretation: Current models appear to internalize preferences as persistent, high‑priority rules rather than conditional behavioral signals contingent on conversational norms and context.

Data & Methods

• Dataset/Benchmark: BenchPreS constructs scenarios that vary (a) the stored user preference, (b) the interaction partner (self vs. third party), and (c) the normative requirement (contexts where applying the preference is appropriate vs. where it should be suppressed).
• Evaluation procedure:
  • Models are given the same stored preference (memory) and are asked to generate responses in multiple contexts.
  • MR and AAR are computed per model across the scenario set to quantify over‑application and correct personalization.
• Models tested: Multiple state‑of‑the‑art LLMs (described generically as “frontier models”); comparisons analyze differences in preference adherence and misapplication.
• Ablations: Experiments include:
  • Varying the strength of preference encoding.
  • Applying reasoning prompts (e.g., chain‑of‑thought) to encourage context reasoning.
  • Adding prompt‑based defensive instructions to suppress preferences where inappropriate.
• Outcomes: Quantitative comparisons show systematic MR even when AAR is high; defenses and reasoning reduce but do not eliminate MR.

Implications for AI Economics

• Externalities and third parties: Misapplied personalized behavior creates negative externalities on third parties (privacy violations, normative harms, misinformation, contractual breaches), which markets and platforms may not internalize without regulation or design changes.
• Trust and adoption costs: If models frequently leak or misuse preferences in third‑party contexts, users and organizations will discount the value of personalization or demand stronger controls, increasing costs for deploying memory features and reducing consumer surplus from personalization.
• Liability and regulation: The failure mode suggests a need for legal and regulatory frameworks that assign liability or require transparency/audits for context‑aware memory systems. Regulatory intervention can change platform incentives to invest in context‑gating mechanisms.
• Platform design and competitive differentiation: Firms offering robust context‑sensitive memory gating (e.g., fine‑grained policy engines, provenance, auditable suppression logic) may capture value by reducing downstream harms and liability—this becomes a potential product dimension and competitive moat.
• Pricing and contracts: Service providers may need to price personalization differently (higher price for guaranteed context safety or separate premium controls) and offer contractual guarantees or indemnities for third‑party harms.
• Mechanism design: BenchPreS provides an evaluative tool for mechanism designers to measure and compare context‑sensitivity, informing incentive structures (e.g., penalties, certifications) that encourage models to treat preferences as conditional signals.
• Research and investment priorities: Economic arguments support investing in technical solutions (contextual memory gating, contextual policy layers, RL with human feedback conditioned on norms) and institutional solutions (standards, audits, certification regimes) to internalize third‑party harms and restore efficient personalization markets.
• Social welfare trade‑offs: There is a trade‑off between personalization value and social/normative risk; optimal policy and product design should balance AAR gains against MR costs, using metrics like BenchPreS to quantify that frontier trade‑off.