Automatic sentiment and emotion classifiers misread men’s text more often than women’s: a study of over one million self-tagged posts finds higher error rates for male authors across 414 model–emotion combinations, including LLM-based tools, implying firms and researchers should not assume off-the-shelf detectors behave equitably when gender composition varies.

Main Finding

Automatic sentiment and emotion classifiers — including dictionary methods, common ML models, and tested large language models — systematically underdetect emotions expressed by men. Across a large, ecologically valid dataset of self-annotated posts, error rates for texts authored by men are consistently higher than for texts authored by women; this bias is large enough in many cases to meaningfully distort downstream analyses that rely on automated affect detection.

Key Points

• Dataset and scope
  • Primary data: TalkLife social platform — authors self-tagged moods. Raw: 6.63M posts by 316k users; analysis sample after filtering: 1.71M posts by 146,883 users with binary gender (Male / Female), 16 mood tags mapped to 5 emotion categories (Sadness, Anger, Fear, Affection, Happiness).
  • Models evaluated: 414 model × emotion/mood combinations spanning dictionary-based lexica, popular ML classifiers, and several open-weight LLMs (LLM evaluation was exploratory, not pre-registered).
  • Two error metrics: valence error (mislabeling positive↔negative) and salience error (classifying emotional posts as neutral).
• Core quantitative findings
  • Men show significantly higher valence errors in all 18 model × sentiment cases (P < 0.001).
  • Men have higher valence errors in 43/45 model × emotion cases (P < 0.001/5).
  • Men have higher valence errors in 115/144 model × mood-tag cases (P < 0.001/16).
  • LLMs tested also display the same direction of bias (higher errors for men) for both valence and salience.
  • Human annotator comparison (enISEAR dataset): annotators also make more errors identifying emotions in men’s text (27.8% vs 23.6%, P = 0.001; majority-label comparison 23.4% vs 18.2%, P = 0.047).
• Downstream impact
  • The authors define a bias-attribution factor k̂: the relative change in detected sentiment that can be explained purely by gender composition differences given differential error rates.
  • k̂ can reach ~50% in some model × mood-tag cases; 48% of combinations have k̂ > 10%, implying substantial risk of confounding in applied analyses (e.g., comparisons across groups or time).
• Robustness and limits
  • Bias appears across model families and at multiple granularity levels (sentiment, emotion, mood tags).
  • Limitations: English-only data, binary gender focus (insufficient data for non-binary categories), restricted set of models (no commercial black-box APIs), zero-shot prompting only for LLMs, mechanisms behind bias not fully identified.

Data & Methods

• Data sources
  • TalkLife (self-reported mood tags + self-declared gender): initial 6.63M posts (2011–2019); filtered to 1.71M posts (5–512 RoBERTa tokens) with one of 16 selected mood tags and known binary gender.
  • enISEAR (controlled dataset) to compare self-reported emotion labels vs third-party annotator labels with known author gender.
• Label mapping and granularity
  • 16 mood tags mapped to 5 emotion categories, and to a 3-way sentiment scheme (positive / negative / neutral) for model output standardization.
• Models and evaluation
  • Evaluated dictionary-based lexica, common supervised ML classifiers, and several LLMs (open-weight, run locally).
  • Standardized model outputs into the 3-way sentiment mapping for cross-model comparison.
  • Error metrics:
    • Valence error: proportion of posts where model returns the opposite valence to the author’s self-label.
    • Salience error: proportion of emotional (non-neutral) self-labels that the model marks as neutral (computed only for models that can return neutral).
  • Statistical tests: χ2 tests with Bonferroni correction per pre-registration; bootstrap resampling for confidence intervals.
• Pre-registration
  • Main design and many analyses were pre-registered; LLM experiments were exploratory and not pre-registered.

Implications for AI Economics

• Measurement validity and bias in economic studies
  • Many economics and policy applications use automated sentiment/emotion measures (consumer confidence, labor market well-being, public response to policy, market sentiment, mental health surveillance). Systematic underdetection of men’s emotions introduces non-random measurement error that can bias estimates and inference.
  • Comparisons across groups or over time are vulnerable: changes in observed sentiment may reflect shifts in gender composition or platform activity rather than true changes in emotions (bias-attribution factor k̂ can be large).
• Potential consequences for applied work
  • Misestimation of population sentiment (e.g., understating male distress or positive affect) can lead to incorrect policy prioritization, mis-targeted interventions, or flawed marketing/consumer-insight decisions.
  • Algorithmic decision systems (e.g., automated triage in mental health services, targeted outreach) risk underdetecting men’s needs, creating unequal access to support.
• Recommended best practices for researchers and practitioners
  • Audit and report: Routinely audit emotion/sentiment classifiers by author demographics where possible; report group-specific error rates.
  • Sensitivity analyses: When comparing groups or time periods, run robustness checks that account for possible gender-dependent error (e.g., reweighting, stratified analyses, or computing k̂ to assess maximum plausible bias).
  • Use ground truth where possible: Prefer self-reported emotion labels (or validation subsets with self-report) when training/validating affect models intended for population inference.
  • Calibration and mitigation: Consider fairness-aware reweighting or calibration, augment training data to improve representation of underdetected groups, and incorporate demographic covariates where ethically and legally appropriate.
  • Transparency about limits: Explicitly acknowledge the uncertainty introduced by differential detection accuracy—especially when gender composition is unknown or changing.
• Research and policy directions relevant to AI economics
  • Construct and share diverse datasets with self-reported affect across demographic groups to enable better model training and validation for economic applications.
  • Investigate mechanisms (linguistic expressivity vs annotator/labeling practices vs model training bias) to design targeted fixes.
  • Evaluate commercial and larger closed-weight LLMs for similar biases, since the present study was limited to open-weight models.
  • Develop standardized bias-attribution diagnostics (like k̂) for use in economic analyses that rely on automated affect measures.

Summary recommendation: If you use off-the-shelf sentiment/emotion classifiers in economic research or applications, treat their outputs as potentially gender-biased measurements. Validate against representative self-reported data where you can, stratify or correct for gender-dependent errors, and perform sensitivity checks reporting how much observed effects could be driven by detection bias rather than true differences in emotional expression.