Data‑driven persona‑conditioned LLM agents reliably predict whether A/B test variants win in an e‑commerce benchmark—achieving 75–90% directional accuracy—and public e‑commerce personas can rival proprietary pools while subsampling halves simulation cost without measurable loss.

Main Finding

LLM agents conditioned on data-driven, behaviorally grounded personas can predict A/B test directions with high directional accuracy (0.75–0.90 depending on metric). The best configuration (pairwise rating questions + domain-aligned personas + appropriately composed persona pool) yields reliable sign predictions, enabling low-cost pre-screening of experiments and a ~2× inference-cost reduction via persona subsampling without measurable loss in directional quality.

Key Points

• Task framing: A/B test simulation is cast as a structured question-answer task where each persona-conditioned LLM rates control and treatment variants; per-persona scores are aggregated into a predicted treatment-effect distribution and evaluated against ground-truth directional labels.
• Best question format: Pairwise rating (present both variants to the agent and ask for a 1–10 score) substantially outperforms independent and/or binary formats (pairwise rating: CTR Acc ≈ 0.75, Subscriptions Acc ≈ 0.80).
• Persona construction:
  • Personas are synthesized by an LLM from anonymized, aggregated behavioral signals (inferred demographics, engagement stats, narratives).
  • Domain alignment matters: in-domain (e-commerce) persona sources produce the best results. Public open e-commerce personas can rival proprietary platform personas (e.g., subscription tests: open e-comm 0.90 Acc vs. proprietary 0.80).
• Depth vs diversity trade-off:
  • Deep pool (highly active users → richer persona histories) outperforms representative pool on CTR (0.75 vs. 0.60 Acc) but not on subscription tests (both ≈0.80 Acc). Behavioral depth helps for engagement metrics; demographic coverage can compensate for sparser per-persona data on intent metrics.
• Subsampling: Selecting a reduced persona subset (e.g., 500 from 935) via simple strategies (random, Kernel Herding, Greedy Farthest) preserves accuracy; authors report a 2× cost reduction with no measurable quality loss.
• Evaluation focuses on sign-based metrics: directional accuracy, sign overlap, and Sign Bhattacharyya distance—prioritizing correct decision direction over exact effect-size estimation.
• Implementation details: experiments used Claude Sonnet 4.5, temperature 0, JSON-constrained responses; benchmark comprised 40 curated A/B tests (20 CTR, 20 subscriptions).

Data & Methods

• Benchmark: 40 curated marketing/e-commerce A/B tests evaluating visual variants; ground-truth directional labels derived from historical experimental confidence intervals (thresholds tuned per metric).
• Persona pools:
  • Proprietary platform-derived pools: Deep (935 high-activity personas) and Representative (935 stratified personas).
  • Public-source pools (each 1000 personas): social-survey, movie-ratings, open e-commerce.
• Persona synthesis pipeline: anonymized behavioral signals → LLM → structured persona (demographics, behavioral stats, narratives).
• Simulation pipeline:
  • Each persona sees visual stimuli for control vs. treatment; pairwise rating prompt elicits numeric scores + free-text reasoning.
  • Aggregate across personas to get predicted effect distribution N(δ̂s, σ̂s).
• Subsampling algorithms tested: uniform random, Kernel Herding (MMD-based greedy selection), Greedy Farthest (diversity maximizing).
• Statistical tests: one-sided paired t-tests (α = 0.05) used to compare configurations.
• Limitations noted by authors: curated benchmark (not fully representative), persona generation biases (sparsity handling, demographic calibration), LLM systematic biases (positional, sycophancy), and focus on directional rather than magnitude alignment.

Implications for AI Economics

• Lowering experimentation costs and increasing throughput:
  • Reliable pre-screening with persona-conditioned LLMs can reduce the number of full live A/B tests run, saving user traffic, engineering time, and calendar time. This increases idea throughput and accelerates product iteration, altering return-on-investment calculus for product development.
• Reallocation of R&D resources:
  • Teams may shift budget from running many marginal live tests to investing in richer simulation tooling (persona generation, LLM inference infrastructure), or reallocate engineering/analytics effort toward variants flagged as promising by simulations.
• Labor and market effects:
  • Preliminary experiment evaluation could be automated, reducing routine analysis workloads for experiment designers and analysts; demand may grow for specialists who tune simulation pipelines and correct for biases.
  • New service markets may emerge (persona engineering, simulation-as-a-service) and existing A/B testing consultancies may adapt to offer LLM-based pre-screening.
• Value of domain data and data externalities:
  • Domain-aligned behavioral data substantially improves simulation fidelity. Platforms with rich behavioral logs gain comparative advantage in internal simulation capability. Conversely, public-domain e-commerce datasets can democratize some capabilities.
  • There is a potential competitive externality: firms that share or publish relevant behavior data (or models trained on it) could indirectly raise simulation quality across the industry.
• Risks of bias and misallocation:
  • If simulations systematically prefer variants favored by over-represented personas (e.g., power users) or reflect LLM biases, firms may prioritize changes that benefit a subset of users, introducing welfare and fairness concerns. Decisions driven by simulated pre-screening require careful calibration and later live confirmation.
• Strategic and regulatory considerations:
  • Widespread use of simulation for product decisions could create feedback loops (simulations trained on prior outcomes influence which experiments are run, shaping future data), potentially entrenching incumbent designs or user segments.
  • From a regulatory perspective, using generated personas and LLM-based behavioral predictions in product design may raise transparency and accountability questions, particularly for user-impacting interfaces (e.g., financial, health-related features).
• Research & investment priorities:
  • Economic value accrues to improvements in persona quality (better sparse-data handling, demographic calibration), robust evaluation metrics beyond direction (magnitude calibration), and cost-efficient subsampling/embedding techniques.
  • Firms should invest in validation pipelines tying simulation outputs to real experiment outcomes to quantify true cost savings and decision-quality impacts before wide deployment.

Suggested next steps for economists and decision-makers: - Conduct cost–benefit analyses comparing simulation-driven pre-screening workflows vs. current A/B pipelines (including false positives/negatives costs). - Audit simulations for representativeness and fairness, and add calibration steps or constraints to prevent systematic misallocation of product changes. - Pilot integration: use simulations to triage candidates, but continue to require live A/B confirmation for deployment-critical decisions.