AI-written and human-edited AI marketing emails substantially boost short-run sales: randomized trials at one online retailer show LLM and hybrid emails roughly double gross order profits versus no-email, and a profit-based decision rule prefers AI options to conventional human writers after accounting for labor and software costs.

Paper: Dubé, J.-P. & Xu, A. (2026). Large Language Models and Creative Content Design: a case study of email marketing at Wine Access. Quantitative Marketing and Economics (24:1). https://doi.org/10.1007/s11129-025-09303-9

Main Finding

In a three-stage randomized controlled case study at a small online wine retailer, LLM-generated email newsletters—either used directly or in a hybrid (LLM + human editing) workflow—produce roughly the same gross profits per campaign as newsletters written by a salaried writing team, while materially reducing annual operating costs. Under a frequentist minimax-regret decision rule (asymptotic minimax regret), one of the AI policies is chosen over the full-human policy in each experiment, implying that AI implementation (automation or augmentation) is the profit-maximizing choice for this firm.

Key Points

• Experimental design: three randomized field experiments comparing four cells (no-email control, Human writers, LLM-generated, Hybrid LLM→human edits). Later experiments varied who edited the hybrid output (professional writer vs. marketing head) and switched from a pre-trained LLM to a prompt-based GPT (Claude).
• Main outcomes: all three active email cells (Human, LLM, Hybrid) approximately doubled purchase incidence, revenues, and gross profits relative to the no-email control. Differences in gross profit between Human and AI cells were statistically insignificant at pre-registered levels.
• Cost effect: because AI policies substantially reduce writer labor costs (with modest software license fees), decision-theoretic analysis selects at least one AI policy over the Human policy in every RCT when considering annual net profit.
• Productivity heterogeneity: evidence suggests salaried writers’ productivity declined under AI automation, while marketers receptive to AI experienced productivity gains under hybrid workflows.
• Contribution: extends prior LLM email-subject-line RCTs by testing LLMs on full email creative and evaluating financial outcomes (gross profits) and normative firm decisions using frequentist decision theory.

Data & Methods

• Setting: Wine Access (WA), a Napa-based online wine retailer that sends twice-daily newsletter campaigns (NC). Unit of analysis: a "campaign" (twice-daily NCs over two weeks).
• Samples & power: WA allocated a sampling frame of ~27,500 regular customers per experiment. Decision-theoretic (AMMR) framework implies much smaller sample requirements than conventional hypothesis testing; several thousand consumers per cell suffice for useful decisions.
• RCTs:
  • RCT1 (early 2024): 4 cells — Control (no NC), Human (3-person salaried writing team), LLM (pre-trained LLM via GUI), Hybrid (LLM output edited by a salaried writer not in the Human cell).
  • RCT2: Hybrid editing shifted to the marketing team (more AI-positive editors).
  • RCT3 (2025): switched from pre-trained LLM to prompt-based GPT (Claude); prompts included wine attributes plus an example historical offer; no historic-email training dataset used in this trial.
• LLMs & implementation:
  • Pre-trained LLM: Mistral7B trained on five years of WA historic newsletters (3,148 unique NCs, 2019–2023) via a third-party consultant and delivered via a GUI that generated draft NCs in ~5 seconds.
  • GPT (Claude): prompt-based generation requiring more prompt effort; chosen for readability and lower hallucination rate; licensing fee differences noted.
• Economic inputs / costs:
  • Current human-writing cost: three salaried writers at $125,000 each ($375,000/year).
  • Pre-trained LLM license: ~$1,000/year; adopting LLM automation could reduce writer labor by ~91.67% (retaining one part-time writer).
  • Claude license: ~$1,200/year; different labor savings under automation/hybrid (83.33% or 75% depending on configuration).
  • Typical human NC writing time: ~3 hours; LLM GUI produced drafts in seconds (but GPT prompts required more editing).
• Decision theory:
  • Frequentist minimax-regret approach (asymptotic minimax regret, AMMR) used to choose policies based on estimated treatment effects and annual net profits (gross profits from orders minus labor and license overhead).
  • The optimal test decision reduces to threshold rules on estimated treatment effects; threshold examples computed relative to break-even annual costs (e.g., a per-consumer breakpoint of ~$0.35 in an illustrative comparison).
  • Advantage: actionable policy recommendations with realistic small-sample sizes for small firms; does not depend on p-values or standard errors in the same way null-hypothesis tests do.
• Outcomes:
  • Treatment effects measured over 2-week campaigns; all three active cells produced statistically and economically significant increases over no-email control.
  • No robust statistical difference between Human vs. AI cells in gross profits across experiments.
  • Decision-theoretic selection favored AI in all experiments after accounting for license and labor cost differences.

Implications for AI Economics

• Automation vs. augmentation: This case illustrates scenarios where AI can substitute for routine creative labor (email copywriting) without loss of revenue—creating strong incentives for labor displacement in small digital-content tasks. Hybrid workflows can yield productivity gains for staff who are receptive and adept at working with AI.
• Cost-effectiveness for small firms: With modest license fees and substantial labor cost reductions, LLMs can be profit-increasing for small firms even when performance (gross profit per campaign) is roughly equal to human-generated content. Decision-theoretic methods make rigorous adoption decisions feasible with small samples.
• Heterogeneous effects and labor markets: The heterogeneous productivity effects (decline for salaried writers; increase for receptive marketers) imply that AI adoption will have distributional consequences—differential demand for editing/supervisory skills and possible job reallocation toward monitoring, prompt design, and fact-checking.
• Research and policy implications:
  • External validity caution: results are from a single firm, product category (wine), and unpersonalized newsletter creative; outcomes may differ for other product types, highly personalized targeting, or firms that already employ individualized creative strategies.
  • Quality control and hallucinations: even with good empirical outcomes, LLMs occasionally hallucinate or make factual errors; firms must invest in editing/fact-checking protocols, which affects net labor savings.
  • Dynamic effects: as LLMs and prompt-based GPTs evolve, the comparative advantages, editing needs, and cost structures will change; continuous evaluation (via small RCTs + decision theory) is advisable.
  • Labor policy: evidence of substitutability in routine creative tasks reinforces arguments for worker reskilling, safety nets, and policies to smooth transitions for displaced creative workers.
• Methodological implication for applied micro studies: the paper showcases AMMR/frequentist decision-theory as a practical framework for small firms to make data-driven adoption choices when conventional large-sample hypothesis testing is infeasible.

Limitations to note: single-firm case study (generalizability limits), some decision-theoretic approximations (AMMR multi-arm extension not solved), assumed symmetric weight on Type I/II errors, and the campaign-level (not per-email) treatment effect assumption.