Telling users an LLM is biased against their party cuts its ability to correct economic-policy misconceptions by 28%; warnings change interactions — users push back more and engage less receptively.

Main Finding

A short message implying that a conversational LLM (ChatGPT 4.1) is biased against a respondent’s party substantially reduces the model’s ability to correct political/economic misconceptions. In a preregistered U.S. survey experiment (N = 2,144), the strongest out-party bias cue attenuated persuasion by 28% (95% CI: 16%–40%) relative to a no-information control; a lighter out-party cue produced a 23% attenuation (95% CI: 9%–35%).

Key Points

• Experiment structure
  • Four-arm, between-subjects, preregistered design: (1) no-information control, (2) non-directional bias info, (3) out-party bias — light, (4) out-party bias — heavy.
  • Respondents engaged in a three-round ChatGPT (non-reasoning 4.1) conversation about one of six common economic-policy misconceptions they personally held.
• Persuasion effects
  • Control group average pre→post change in misconception agreement: −1.20 (0–4 scale).
  • Heavy out-party cue group change: −0.86 → 28% reduction in persuasion vs control.
  • Reversal rates (full switch from at-least-moderate misconception to no agreement or consensus): control 34.4% → heavy treatment 22.1%.
• Mechanism and interaction dynamics
  • Transcript analysis shows respondents in out-party cue conditions wrote more and were more argumentative (not simply disengaged or dismissive), consistent with directional motivated reasoning.
  • Manipulation checks: both out-party treatments increased perception that ChatGPT favored the opposing party for respondents of both parties, eliminating baseline partisan differences in perceived bias.
• Heterogeneity and robustness
  • Effects persist across topics and across partisanship; no statistically significant heterogeneity by party, affective polarization, prior trust in AI, or self-reported topic knowledge.
• Additional outcomes
  • Out-party cues reduced perceived persuasiveness of the LLM, willingness to use AI again for opinion challenge, general trust in AI chatbots, and support for politicians using AI for policy information.
• Limitations noted by authors
  • Single, short one-off interaction; single model family (non-reasoning ChatGPT 4.1); U.S. online sample. Results may differ with longer exposure, different models, or other populations.

Data & Methods

• Sample and timing
  • Fielded Dec 2025–Jan 2026. Initial respondent pool N = 2,414; final experiment N = 2,144. 93% of initial pool held at least one targeted economic misconception.
• Treatment
  • Control vs non-directional bias information vs two out-party bias messages (light and heavy). Heavy treatment included cues about founder political ties and an image implying out-group alignment.
• Conversation
  • Three-round prompts to ChatGPT 4.1: model given the respondent’s answer and instructed to persuade toward economist consensus while remaining truthful.
• Outcomes
  • Primary: change in agreement with a chosen misconception (0–4 scale) pre- and post-conversation; reversal to consensus.
  • Secondary: perceived persuasiveness, willingness to reuse AI, general trust in AI, support for politician use of AI.
• Transcript scaling and mechanism tests
  • Open-ended chat transcripts were compared using an LLM-as-judge pairwise method (≈30 comparisons/transcript) and scaled with a Bayesian Bradley–Terry model to measure argumentative and dismissive behaviors.
• Statistical analysis
  • OLS models of post-treatment agreement controlling for pretreatment agreement and topic fixed effects. Bootstrap (stratified) used for CIs on attenuation percentages. Multiple robustness and heterogeneity checks reported.

Implications for AI Economics

• Credibility is an economic input: perceived political neutrality materially affects the marginal effectiveness (benefit) of LLM-based persuasion or corrective information. Perceived bias reduces the return on deploying conversational LLMs for public information campaigns.
• Demand and adoption effects
  • Reduced willingness to reuse AI and lower trust imply lower sustained demand for LLM-based advisory/fact-checking services among groups primed to see bias. This creates heterogeneity in adoption and use-value across political segments.
• Strategic externalities and incentives
  • Partisan actors can reduce the public-good value of LLMs by credibly or cheaply signaling bias (e.g., media claims, political messaging). This suggests a negative externality: third-party attempts to manipulate perceived neutrality can lower the social returns of information-correcting technology.
  • There is a potential arms race: AI providers and public-interest actors may need to invest in alignment, transparency, audits, or reputation-building to protect persuasive value — these are costly investments whose returns depend on reducing perceived bias.
• Cost-effectiveness of AI interventions
  • If perceived bias increases the cognitive resistance (more argumentative responses) required to change beliefs, the marginal cost (effort, time, content tailoring) of persuasion rises. Economic evaluations of LLM interventions should incorporate credibility-framing and counter-messaging costs.
• Market design and product strategy
  • Firms may have incentives to (a) design interfaces and messaging that mitigate perceived partisan alignment (credibility signaling), (b) offer calibrated products for ideologically heterogeneous markets, or (c) monetize credibility (third-party certification).
  • Political economy of AI investment: returns to alignment and neutrality-preserving R&D are higher when political actors actively attack neutrality; investors and regulators will factor this into valuations and policy choices.
• Policy and regulatory implications
  • Regulators interested in maximizing public-information benefits might promote standards for transparency, independent audits of political bias, or disclosure regimes to stabilize perceived neutrality — reducing the effectiveness of partisan credibility attacks.
• Modeling and future research directions for AI economists
  • Incorporate perceived-source credibility as an explicit state variable in diffusion/persuasion models (e.g., Bayesian updating with a credibility prior or a two-stage game where actors invest in shaping perceived bias).
  • Analyze equilibrium where political actors allocate resources to (i) discredit LLMs, (ii) invest in pro-LMM reputation, and (iii) use LLMs for persuasion — derive implications for welfare and information quality.
  • Quantify cost-benefit tradeoffs of alignment investments: how much must platform/firm spending on transparency/audits increase to offset persuasion losses from politicization?
  • Empirically: estimate long-run effects across repeated exposures, different model families (reasoning vs non-reasoning), and in non-U.S. or less polarized contexts to generalize welfare and market outcomes.

Suggested concise takeaway for AI economists: perceived partisan bias is a key determinant of the effectiveness and market value of LLM-based information interventions. Political attacks on neutrality can materially lower the social and private returns to deploying LLMs for persuasion or fact correction, creating incentives for alignment, transparency, and regulatory responses — all of which carry costs that should enter economic assessments and models.