GPT-style models match supervised classifiers in labeling U.S. news ideology, and AI explanations boost human confidence and can sway judgments, with detailed explanations proving most persuasive.

Main Finding

With careful prompt engineering, GPT-class LLMs can match state-of-the-art supervised models on multi-class political-ideology (Left/Center/Right) news classification. However, when LLM outputs are used to assist human annotators, AI-generated explanations substantially raise annotator confidence (p < .001) and detailed explanations are more persuasive — they lead to more decision changes but also sometimes mislead annotators. Thus LLMs are promising for scaling news-ideology annotation, but explanation format strongly shapes human reliance and can introduce risks of overreliance and bias amplification.

Key Points

• Research questions:
  • RQ1: How well do LLMs perform on multi-class news ideology classification?
  • RQ2: How do LLM-generated explanations affect human judgment and confidence?
  • RQ3: What motivates users’ trust or distrust of LLM outputs in this subjective task?
• Model performance:
  • With prompt engineering, GPT variants (including GPT-4o) achieved performance comparable to supervised SOTA on a 34k public news dataset (evaluation used F1, accuracy, MAE, MCC).
• New dataset:
  • Authors collected 17,166 recent U.S. news articles (through July 2024) for analysis and human experiments; dataset is being shared for further research.
• Human study:
  • Between-subjects MTurk study with N = 124 participants.
  • Two conditions: brief vs. detailed LLM-generated explanations accompanying the model’s label.
  • Participants labeled articles (Left/Center/Right) and reported confidence before and after receiving AI information.
  • AI assistance increased self-reported confidence irrespective of whether the AI was correct.
  • Detailed explanations led to more label changes (i.e., were more persuasive) but also sometimes induced incorrect switches.
• Explanation design:
  • The team tested multiple prompt templates (simple label-only prompts, expert-role prompts, prompts including indicator lists and media-source priors, and re-analytic prompts encouraging critical re-evaluation).
  • Brief explanations were less likely to change decisions; detailed explanations contained more persuasive signal but higher risk of misleading.
• Qualitative/thematic analysis:
  • The authors analyzed why participants accepted or rejected AI suggestions, identifying themes (e.g., perceived expertise of AI, alignment with user priors, reliance on surface cues, skepticism about source/context).
• Contributions:
  • Demonstrates LLMs’ capability in a highly subjective classification task.
  • Provides the first systematic comparison of brief vs. detailed LLM explanations on annotator confidence, reliance, and decision change in political news labeling.
  • Offers design recommendations for AI explanations and releases a large news dataset.

Data & Methods

• Datasets:
  • Historical 34k public news dataset used for model benchmarking (reference in paper).
  • Newly collected 17,166 U.S. news articles (up to July 2024) for GPT predictions and human experiments.
• Models and prompts:
  • GPT family models (GPT-4o reported) used with prompt engineering. Multiple prompt styles were evaluated:
    • A–D: label-only and expert-role prompts; prompts that list bias indicators and media-source priors.
    • E1/E2: explanation-returning prompts and a “be critical / re-evaluate” prompt.
• Evaluation metrics:
  • Standard classification metrics: F1, accuracy; plus MAE and Matthews correlation coefficient (MCC).
• Human experiment:
  • MTurk recruitment (N = 124), between-subjects design (brief vs. detailed explanation).
  • Participants classified sets of articles, reported confidence on 1–7 scales before and after seeing AI output.
  • Study included quality screening and attention checks; platform: Qualtrics.
  • Analysis combined quantitative measures (confidence shifts, decision change rates, statistical tests — e.g., reported p < .001 for confidence increase) and qualitative thematic coding of participant rationales.
• Analysis:
  • Comparison of GPT performance vs. supervised SOTA on historical labeled data.
  • Behavioral impact measured by changes in labels and confidence, disaggregated by whether AI labels were correct or incorrect and by explanation type.
  • Thematic coding of participant free-text responses to understand reasoning and perception.

Implications for AI Economics

• Labor substitution vs. augmentation:
  • LLMs that match supervised classifiers can reduce marginal annotation costs and speed label generation, potentially shifting demand from cheap crowd annotation toward fewer higher-skilled verification tasks. This can compress prices for routine annotation labor while increasing demand for oversight and quality control roles.
• Productivity and cost structure:
  • Using LLMs to pre-label or suggest labels can raise throughput and reduce time-per-item, lowering annotation budgets for firms and researchers. However, increased need for human verification of explanations (especially detailed ones that may mislead) introduces new verification costs and coordination tasks.
• Quality, incentives, and market signaling:
  • AI explanations increase annotator confidence even when wrong, creating risk of systematic label errors that could bias downstream models, analytics, and products. Market actors relying on cheaper AI-assisted annotation may externalize misinformation or bias, imposing social costs (reputational, regulatory, or consumer trust losses).
  • Platforms and buyers must redesign incentives: reward annotators for independent critical evaluation, not just agreement with AI suggestions. Systems should preserve disagreement as a signal rather than collapsing toward AI consensus.
• Task design and wages:
  • Tasks that remain highly subjective (like media bias) will still require human judgment; platforms may differentiate task tiers (AI-assisted rapid labeling vs. adjudication-level annotation), affecting wage structures and skill premiums.
• Product design and adoption:
  • Explanation format matters economically: detailed explanations are more persuasive and thus more impactful on human decisions — useful for productivity but riskier for correctness. Designers should balance persuasiveness and calibration, and disclose uncertainty to avoid overreliance.
• Externalities and regulation:
  • Misleading AI explanations can amplify polarization and misinformation, creating negative externalities that could spur regulatory scrutiny (transparency, auditability, responsibility for downstream harms). Firms face potential compliance costs and reputational risk.
• Research and public goods:
  • Public release of the 17k dataset lowers research costs and helps build better evaluation benchmarks for AI-human collaboration in subjective domains — increasing social value capture through improved models and policy guidance.
• Practical recommendations for economic actors:
  • Adopt AI-assistance to scale annotation but mandate human-in-the-loop verification for subjective labels; use uncertainty-aware explanations (show confidence/calibration).
  • Monitor label distributions for shifts after AI-assistance and preserve annotator disagreement metrics as quality signals.
  • Invest in training annotators to critically evaluate AI outputs; compensate adjudication work at a premium.
  • When deploying AI-labeled data in downstream products, run robustness checks to avoid amplifying systematic AI biases into market-facing decisions.

If you want, I can: - Extract the paper’s experiment results into a concise table (accuracy/F1/MCC if you provide the numeric values). - Draft a short checklist for deploying LLM-assisted annotation workflows that balances cost savings and risk mitigation.