Sentiment judgments of short, context-free team messages depend more on the message than the rater: individuals' labels are only moderately stable and ambiguous wording drives most disagreement, while mood traits produce only small positive biases. The result warns that automated sentiment signals—often trained on pooled human labels—can be noisy and systematically biased unless annotator and item heterogeneity are explicitly modeled.

Main Finding

Sentiment perception of short, decontextualized messages in team-based software projects is only moderately stable within individuals and is strongly statement-dependent. Individual mood traits (especially trait mood and reactivity) weakly predict a tendency to label statements as more positive (and less neutral), but overall effect sizes are small, negative-label predictors are weaker, and many apparent correlations do not survive correction for multiple testing. Ambiguity in statements drives most label changes; no clear project-phase effects were found.

Key Points

• Study design: four survey rounds with 81 student participants working in teams; each round participants reported mood, life circumstances, project-phase and group-dynamics measures and labeled 30 decontextualized statements for sentiment (positive/neutral/negative), recording rationale and uncertainty.
• Stability: Sentiment labels are only moderately stable within individuals across rounds; label changes concentrate on statements judged as ambiguous.
• Predictors:
  • Trait-level mood and emotional reactivity correlate with more positive (and fewer neutral) labels in statement-level repeated-measures models (GEE).
  • Predictors of negative labeling are weak and at best trend-level (e.g., task conflict).
  • Correlation-level signals are generally small and fail to remain significant after global multiple-testing correction.
• Context effects: No clear evidence that project phase systematically shifts sentiment perception.
• Statement dependence: The particular statement’s ambiguity/wording is a dominant source of labeling variability.
• External validity caution: Study was conducted with students and decontextualized messages; findings motivate replication in real project communication.

Data & Methods

• Sample: 81 students in team-based software projects, surveyed across four rounds (longitudinal repeated-measures design).
• Tasks: In each round, participants labeled the sentiment of 30 decontextualized text statements; provided meta-data on labeling rationale and self-reported labeling uncertainty.
• Measures collected: mood traits, momentary mood states, life circumstances, team-level dynamics (e.g., conflict), and project-phase indicators.
• Analytical approach:
  • Descriptive stability analyses across rounds.
  • Correlation analyses between sentiment labels and predictors, with multiple-testing correction.
  • Statement-level repeated-measures generalized estimating equations (GEE) to model predictors of positive/neutral/negative labeling while accounting for repeated observations per participant and per statement.
• Key statistical outcomes: Small predictor effects overall; trait mood/reactivity significant in GEE for positive vs neutral; many correlations not robust to multiple-testing adjustment.

Implications for AI Economics

• Measurement error and attenuation: High within-person variability and statement-dependent ambiguity imply noisy sentiment labels. In econometric analyses that use automated sentiment measurements (e.g., linking communication tone to productivity, turnover, or team performance), this noise can attenuate estimated effects and bias inference toward zero unless explicitly modeled.
• Heterogeneous annotator effects: Mood traits and reactivity shift labeling propensity (toward positivity). Classifiers trained on labels pooled from annotators with heterogeneous affective traits may inherit systematic biases. Economists evaluating AI tools should account for annotator heterogeneity (e.g., by modeling rater effects or using hierarchical labeling schemes).
• Classifier evaluation and deployment: Because ambiguity is a main source of disagreement, evaluation should emphasize performance on ambiguous items and measure uncertainty calibration. For high-stakes economic decisions (HR monitoring, productivity signals, automated moderation), relying on context-free sentiment scores is risky—invest in contextualized, in-project training data to improve external validity.
• Policy and firm decisions: Automated sentiment analytics used for managerial decisions, performance-based pay, or market signaling can produce erroneous or unfair outcomes if they ignore label variability and ambiguity. Cost–benefit assessments of deploying such tools should incorporate error rates, heterogeneity across raters, and potential adverse reactions from affected workers.
• Research design recommendations for AI economists:
  • Model measurement error explicitly (errors-in-variables, latent-variable models, or use instrument variables where possible).
  • Use multi-annotator labels with recorded uncertainty and model annotator and item (statement) random effects.
  • Prefer contextualized communication data over decontextualized snippets when estimating economic effects of sentiment.
  • Prioritize collecting additional labels for ambiguous items (active learning) and calibrate classifier uncertainty to downstream economic decisions.
• Generalizability caveat: Findings are from a student sample and decontextualized statements; quantify transfer risk before applying these conclusions to industry settings or macroeconomic inference.