Agentic AI that learns from prior experiments outperforms human-guided design: in two large field trials of prescription messaging the tool-augmented AI produced a message with a 69.8% click-through rate, 6.5 percentage points above baseline, while general-purpose LLMs without access to experimental data failed to identify winners.

Main Finding

Tool-augmented agentic AI that executes code, runs statistical analyses, and reasons through a structured Data→Information→Knowledge→Wisdom (DIKW) pipeline can learn from prior field-experiment data and generate superior behavioral interventions. In two sequential randomized field experiments on prescription SMS messaging (693,139 patient visits total), AI-generated messages from the agentic system produced the best-performing variant (69.8% click-through), outperforming the prior-round human + chatbot designs and frontier LLMs that lacked access to the experimental data. The performance gains derived from grounding in domain-specific experimental evidence and transparent evidence chains, not from raw LLM generative ability alone.

Key Points

• Two-stage megastudy design:
  • Stage 1 (Jun 16–Jul 3, 2025): Human experts + conversational LLM co-designed 13 message variants; 444,691 patient visits.
  • Stage 2 (Aug 25–Sep 8, 2025): Tool-augmented agentic AI extracted principles from Stage 1 and generated 17 new variants (tested alongside 3 Stage-1 baselines → 20 variants total); 248,448 patient visits.
• Primary outcome: click-through rate (CTR) on the SMS link; secondary outcome: post-click authentication.
• Best AI-generated message achieved 69.8% CTR (reported +6.5 percentage points over the Stage 2 baseline in the paper; cited as a 10.3% relative improvement in contexts reported by the authors).
• Effective behavioral principles surfaced by the AI: efficiency framing and professional authority. Ineffective principles in this healthcare context: social proof and reciprocity — indicating limits to transferring general nudging theory across contexts.
• Frontier LLMs without access to the experimental dataset (i.e., using only general training data) could not reliably predict which interventions would succeed — highlighting that proprietary, domain-specific experimental data were the key value source.
• System architecture highlights: code execution for statistical analysis, multi-agent DIKW reasoning layers, and transparent evidence chains linking each design decision back to specific experimental observations.

Data & Methods

• Setting: prescription notification SMS from a large U.S. patient messaging platform. Only message text varied; sender, timing, and link mechanics held constant.
• Sample: 693,139 randomized patient-visit invitations across two stages (Stage 1: 444,691; Stage 2: 248,448). Randomization balance confirmed (χ2 p > 0.05).
• Outcomes:
  • Primary: binary click-through on the SMS link.
  • Secondary: authentication after click (deeper engagement).
• Intervention design comparisons:
  • Baseline: expert + chatbot co-designed messages (Stage 1).
  • Treatment: agentic AI (Stage 2) that:
    • Ingested Stage 1 experimental data,
    • Executed statistical analyses (code-enabled tool use),
    • Employed DIKW-structured agents to infer behavioral principles,
    • Generated alternative message texts with explicit evidence chains tied to the analyzed data.
• Validation: Stage 2 was a randomized field trial of AI-designed messages. The study also evaluated frontier LLMs operating without access to the Stage 1 experimental data; those models failed to predict success.
• Robustness notes: message assignment was randomized and demographic/medical context covariates were recorded to explore heterogeneity (age cohorts, gender, therapeutic category, provider specialty, geography/ADI). The paper emphasizes traceability/auditability of the AI reasoning.

Implications for AI Economics

• Value creation from proprietary data and tooling, not LLM architecture alone:
  • Economic value accrues to organizations that (a) collect large, domain-specific experimental datasets and (b) deploy tool-enabled agentic AI to extract reusable design knowledge. This favors incumbents or platforms with rich customer-experiment histories.
• Moves experimentation from one-shot evaluation to cumulative design learning:
  • Agentic AI can operationalize iterative improvement cycles at scale, increasing the return on each experiment and potentially lowering cost-per-improvement in behavioral interventions.
• Market and product implications:
  • Demand for experiment-management platforms, analytics tooling, and agentic-AI systems that provide auditable evidence chains will grow.
  • There is commercial scope for specialized “design-learning” services that convert experimental data into intervention libraries or reusable design primitives.
• Labor and organizational implications:
  • Human experts remain important (for supervision, ethics, and domain judgment), but routine extraction of design principles and large-scale variant generation can be automated, shifting expert work toward oversight, validation, and higher-order theory building.
• Policy, regulation, and welfare considerations:
  • Transparent evidence chains and auditable DIKW pipelines are crucial for accountability, especially in sensitive domains (healthcare). Regulators and institutions should require traceability for automated intervention design.
  • Care is needed before scaling to outcomes beyond proximal engagement (CTR). Economic benefits (e.g., improved adherence, cost savings) require linking improved engagement to downstream clinical and cost outcomes — an avenue for future evaluation.
• Research and methodological implications:
  • The result cautions against relying on general nudging heuristics; scalable, domain-specific auditing of behavioral theories via agentic AI can reveal which principles transfer and which do not.
  • Future economic research should quantify welfare gains from cumulative experimental learning and study competitive dynamics where data-holding platforms can iterate faster.

Limitations to keep in mind: the paper reports gains in a messaging engagement metric (CTR) rather than downstream clinical outcomes; generalizability beyond prescription messaging and other healthcare populations requires further testing; and ethical/regulatory constraints around automated patient-targeted interventions must be actively managed.