Large language models' ethical advice swings with surface framing: scarcity cues increase approval by 12 percentage points while high-stakes outcomes and fair-process prompts reduce it by about 10–11 points. When multiple contextual cues align, recommendations can shift by more than half the probability scale, indicating advisory outputs are highly sensitive to framing rather than fixed principles.

Main Finding

Frontier LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) show large, systematic sensitivity to contextual framings in organizational ethical vignettes. Specific framing dimensions (resource scarcity, outcome severity, procedural justice) change the probability models endorse ethically consequential recommendations by substantial margins (≈10–12 percentage points each), with cumulative and extreme framing creating up to a ~54 percentage-point swing. These effects are consistent across models and are accompanied by systematic shifts in the ethical rationales models produce (“adaptive rationalization”).

Key Points

• Scope and design: Preregistered factorial experiment generating 5,000 unique organizational vignettes (balanced across industries) and collecting 15,000 model responses (3 models × 5,000 vignettes).
• Analysis sample: 694 responses (4.6%) without clear recommendations were excluded, leaving 14,306 responses analyzed via logistic regression with robust, clustered standard errors.
• Principal effects (relative to baseline endorsement probability):
  • Resource scarcity: +12.0 percentage points (pp); OR = 1.67, 95% CI [1.45, 1.93], p < .001.
  • Outcome severity: −11.3 pp; OR = 0.62, 95% CI [0.54, 0.71], p < .001.
  • Procedural justice (emphasizing fair process): −10.1 pp; OR = 0.66, 95% CI [0.57, 0.76], p < .001.
• Effect sizes comparable to classic human framing results (Tversky & Kahneman ~22pp; McNeil et al. ~18pp) and thus substantively meaningful for organizational decision contexts.
• Cumulative and extreme framing:
  • When multiple dimensions align in ethically unfavorable directions, net effects ≈ 27pp from baseline (range 25–28pp under different interaction assumptions).
  • Maximum-to-minimum framing scenarios produce ~54pp total range—approaching full reversal of recommendations.
• Model-general phenomenon: No significant Dimension × Model interactions—effects were consistent across the three models, suggesting core architectural or training-related sensitivity rather than model-specific quirks.
• Adaptive rationalization (from topic modeling of justification text):
  • Under high resource scarcity, models more often invoke utilitarian reasoning (+6.7 pp).
  • Under high outcome severity, models more often invoke deontological reasoning (+2.4 pp).
  • Under high procedural justice cues, models more often invoke virtue/justice ethics (+4.4 pp).
  • Interpretation: models adjust their ethical framing to match contextually salient constraints rather than applying a single consistent ethical principle.
• Human validation:
  • Crowdworker coding: n = 7,500 responses (one rater each), Fleiss’ κ = 0.71; 81.3% concordance with expert codings.
  • Subject-matter experts: n = 24 experts, 100 vignette pairs each (including controls) → 2,400 comparisons. Experts detected framing-driven differences in 48.9% of pairs (after accounting for an 18.3% baseline false-positive rate) but correctly attributed differences to the manipulated dimensions in only 41.3% of cases. Of detected differences, 58.7% were judged problematic for AI advisory systems.

Data & Methods

• Experimental design: Pre-registered fractional factorial design producing 5,000 unique vignettes varying six framing dimensions:
• Procedural justice
• Outcome severity
• Stakeholder power
• Resource scarcity
• Temporal urgency
• Transparency requirements
• Balanced industry sampling to improve external validity across organizational contexts.
• Models tested: Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro.
• Responses: Each vignette was run through each model → 15,000 total responses; 694 ambiguous/no-recommendation responses excluded (4.6%).
• Outcome variable: Binary endorsement recommendation (model recommends the ethically questionable action vs. not).
• Statistical analysis: Logistic regression with robust and clustered standard errors; reported odds ratios, percentage-point changes (marginal effects), 95% CIs, and p-values. Interaction tests for Dimension × Model were performed.
• Qualitative/interpretive analysis: Topic modeling of justification text to identify ethical frameworks; human coding and validation (crowdworkers and subject-matter experts) to verify framing sensitivity is substantive and not an artifact of the measurement pipeline.

Implications for AI Economics

• Decision-support reliability: The magnitude of framing effects implies LLM-based advisors can be substantially swayed by superficial contextual cues. For organizational economics, this threatens consistent policy recommendations, risk assessments, and allocation decisions—introducing endogenous variability tied to prompt/context rather than underlying fundamentals.
• Governance and auditing: Effective deployment requires mandatory stress-testing across realistic framing variations (a “framing robustness” dimension in audits), standardized evaluation protocols, and regulatory disclosure of model sensitivity to contextual features.
• Organizational design & incentives: Firms relying on LLM recommendations should avoid single-model point estimates for consequential decisions. Use ensembles, counterfactual framing checks, and calibrated decision rules (e.g., decision thresholds invariant to framing) to reduce manipulable variation.
• Market design and strategic behavior: If LLM advice is frame-sensitive, actors (internal or external) can potentially game outcomes by manipulating presentation or context cues. This raises new considerations for incentive-compatible mechanism design and governance structures to deter strategic framing.
• Model alignment and training priorities: Consistent cross-model sensitivity suggests training and alignment procedures must prioritize invariance to irrelevant contextual framing and foster stable ethical reasoning rather than adaptive rationalization that mirrors the most salient cues.
• Cost-benefit trade-offs and regulation: Economists and policymakers should incorporate framing-robustness metrics into cost–benefit analyses of automated decision-support adoption, and consider regulation that mandates robustness testing for high-stakes applications (finance, health, HR, safety-critical operations).
• Research directions: Quantify economic impacts of framing-induced recommendation variability (e.g., error rates, welfare consequences), develop formal measures of "framing elasticity" for models, and test mitigation strategies (prompt engineering standards, constrained decoding, supplementary rule-based checks, human-in-the-loop safeguards).

Limitations to note: laboratory vignette setting may not capture all field complexities; only three models were tested (though effects were consistent across them); 4.6% of outputs were uninterpretable and excluded; attribution accuracy by experts was imperfect—practical detection of framing effects remains nontrivial. These caveats reinforce the need for operational robustness checks in real deployments.