Generative-AI assistants speed up after-sales chat and boost customer ratings overall, but benefits concentrate among lower-performing agents while top agents become more distracted and sometimes worsen outcomes.

Main Finding

Access to a generative-AI assistant in Alibaba’s after-sales chat support causally increased agent speed and raised customers’ subjective satisfaction, but did not improve objective resolution rates on average. Effects were heterogeneous: low-performing agents gained the most on both speed and perceived quality (narrowing the performance gap), while top-performing agents experienced declines in both subjective and objective quality driven by increased multitasking.

Key Points

• Intervention: Agents were randomized to access a gen-AI assistant that produced two text suggestions per chat (issue diagnosis and solution); agents could copy, edit, or ignore suggestions.
• Aggregate effects (ITT / LATE):
  • Faster service: shorter issue-identification time and reduced chat duration.
  • Higher subjective quality: increased customer ratings and lower dissatisfaction rates.
  • No significant improvement in objective quality overall: customer retrial rates (a proxy for unresolved issues) were unchanged on average.
• Mechanisms:
  • Agent behavior: responses became faster, more numerous, and linguistically more informative (higher lexical density, clarity, diversity).
  • Customer behavior: reduced communication burden (fewer supporting pictures uploaded, simpler language), implying improved perceived clarity/effort.
  • Both automation (time savings) and altered interaction dynamics (more informative agent messages, less customer effort) drove improvements in perceived service.
• Heterogeneity by pre-treatment performance:
  • Low performers (bottom quintile) showed the largest improvements in speed and subjective quality.
  • Top performers (top quintile) showed little speed gain and a deterioration in both subjective (ratings, dissatisfaction) and objective (retrial) outcomes.
  • For top performers, process data indicate increased shift-away times and multitasking across concurrent chats, slowing customer responses, increasing abandonment, and triggering immediate retrials.
• Adoption and usage:
  • Average adherence to AI suggestions was low and varied by skill level; the authors estimate causal LATE of actual AI usage in addition to ITT of access.
  • Top agents were more skeptical and used AI suggestions less frequently, often verifying outputs more.

Data & Methods

• Setting: Large-scale randomized field experiment in Alibaba/Taobao after-sales chat support (order-related returns/exchanges/refunds/repairs).
• Randomization: Agents randomly assigned to treatment (AI access) or control (no AI access); assignment remained as intended.
• Intervention: A gen-AI assistant (Alibaba’s Qwen LLMs fine-tuned on service chats and order data) provided two text suggestions per chat: (1) diagnosis, (2) proposed solution. Suggestions shown in chat sidebar; agent discretion preserved.
• Outcomes measured from fine-grained process logs and post-chat surveys:
  • Service speed: issue identification time, total chat duration, response latency.
  • Subjective quality: customer post-chat rating (1–5), dissatisfaction indicator.
  • Objective quality: customer retrial rates (repeat contacts), abandonment rates.
  • Interaction measures: number of messages, lexical density/clarity/diversity of agent text, customer uploads (images), shift-away time (time agents spend away from a chat while handling others).
• Identification strategy:
  • Intention-to-treat (ITT) estimated the effect of access to AI.
  • Local average treatment effect (LATE) estimated causal effect of actual AI usage (instrumenting usage with assignment).
  • Heterogeneity analyzed by pre-treatment performance quintiles.
• Supplementary evidence: textual analysis of messages, process-level timing metrics, and internal agent survey data used to probe mechanisms (use patterns, attitudes).

Implications for AI Economics

• Human-AI complementarity is nuanced: gen-AI can speed up tasks and improve perceived service without necessarily improving objective task success. Economists should distinguish productivity gains in speed/effort from gains in underlying task effectiveness.
• Distributional effects:
  • Gen-AI can reduce within-firm performance dispersion by disproportionately helping low performers — a potential equalizing force that may affect returns to skill and wage dispersion in service sectors.
  • Conversely, it can harm top performers’ realized performance through workflow changes (e.g., induced multitasking), suggesting non-monotonic effects across the skill distribution.
• Behavioral channels matter: productivity effects arise through changes in interaction dynamics (message informativeness, customer effort), not only automation. Modeling frameworks should incorporate behavioral responses (multitasking, verification, trust/aversion) to AI tools.
• Measurement caution: reliance on subjective satisfaction metrics alone can overstate welfare gains if objective resolution remains unchanged. Policy and management evaluations should include objective outcomes (retrial/resolution rates) and process-level indicators.
• Managerial and policy recommendations:
  • Tailor deployment: prioritize AI support for lower-skilled workers to maximize gains and reduce variance.
  • Monitor unintended workflow effects: design UI and incentives to limit harmful multitasking (e.g., discourage excessive shift-away behavior, prioritize single-chat focus when needed).
  • Training and role design: provide training to integrate AI suggestions productively and preserve tasks that keep top performers engaged, avoiding demotivation or deskilling.
  • Performance metrics: combine speed, satisfaction, and objective resolution in KPIs to avoid perverse incentives (e.g., rewarding fast but ineffective handling).
• Research directions: study longer-run effects on skill development and labor outcomes (wages, retention), general equilibrium impacts across firms/sectors, and design interventions (interface, incentives, verification protocols) that mitigate adverse effects for high performers while amplifying benefits for low performers.