Synthetic buyers powered by large language models reproduce real A/B-test outcomes on a major e-commerce platform, cutting experiment turnaround from weeks to under an hour and enabling rapid offline experimentation without risking real customers.

Main Finding

SimGym is a production-scale system that uses LLM-driven browser agents, grounded in shop-specific traffic data, to run offline A/B tests of e‑commerce UI changes. On a held-out set of 20 real store theme changes (confounder-controlled), SimGym agents predict the direction and magnitude of observed add-to-cart (A2C) changes with substantial accuracy (Pearson correlation ≈ 0.65 once agent sample size is sufficient) and sign‑alignment that increases with agent count (≈51% at 50 agents → ≈73% at 700 agents). The authors show this can compress experiment cycles from weeks to under an hour and enable pre-testing of UI changes without exposing real customers.

Key Points

• Novelty
  • First reported evaluation of LLM-based synthetic shoppers that is validated against real human A/B outcomes in production e-commerce.
  • Agents operate in a live browser (accessibility tree interface) rather than on static datasets, requiring real‑time perception, memory, and action execution.
• Persona and intent grounding
  • Per-shop persona pipeline extracts buyer intents and multi-dimensional behavioral/value traits from clickstream data (six-stage pipeline: session clustering → product preferences → intent generation → buyer aggregation → persona construction → prompt composition).
  • Persona axes include price sensitivity, exploration depth and values (premium / performance / ethics).
  • Intents are calibrated to cluster-level A2C rates (mix of purchase-ready vs research intents).
• Agent architecture
  • Perceive–plan–act loop using an accessibility tree for page state; schema-constrained LLM responses (reasoning + action); episodic session memory carrying the full session log.
  • Guardrails: loop/infinite‑loop protection, step/time limits, retry logic, and error propagation.
• Ground truth and evaluation
  • Ground-truth dataset built from real theme changes filtered to remove confounders (seasonality, promotions, assortment/pricing shifts) and validated with double machine learning; final sample = 20 shops across 12 countries.
  • Primary metric: A2C rate change. Metrics: alignment rate (directional agreement), alignment probability (Bayesian posterior of direction), and Pearson correlation for magnitude across buyer clusters.
  • LLM evaluator flags simulation failures (loops, intent mismatches, out-of-stock attempts) to avoid metric inflation.
• Empirical results & operational choices
  • Bootstrap analysis of agent sample size showed diminishing returns; Pearson correlation stabilizes around 0.65 after ~300 agents and sign‑alignment improved up to ~700 agents. Authors choose 600 agents/shop as a practical operating point.
  • Ablations indicate episodic memory and per-shop persona grounding materially contribute to predictive validity (removing them degrades alignment and correlation).
• Limitations acknowledged
  • Ground-truth set is modest (20 shops) and restricted to theme changes detectable in the DOM (visual-only effects require vision-enabled agents).
  • Simulations rely on historically grounded personas and LLM behavior; potential gaps when UI changes elicit novel human behaviors not present in the history.

Data & Methods

• Data sources
  • Production clickstream data per shop: session-level features (duration, event count, product views, searches, funnel progression A2C/checkout/purchase, cart/order values).
  • Ground-truth: observed pre/post theme-change human behavior for shops passing confounder filters.
• Persona & intent pipeline (six stages)
• Session-level clustering (k-means, k=5) over standardized session features to identify behavioral clusters.
• Product preference extraction: LLM summarizes browsed/purchased products into categories and popular items per (shop, cluster).
• Buyer intent generation: calibrated counts of purchase-ready vs research intents (purchase count = clip(round(mean A2C * n),1,n−1)); strict two-sentence intent templates; constraints to avoid UI/size/discount leakage.
• Buyer-level aggregation: select n centroid-proximate sessions and aggregate to buyer grain (funnel rates, interaction histories).
• Persona construction: continuous scores for behavioral (price sensitivity, exploration) and value (premium, performance, ethics) axes; LLM outputs structured JSON with scores and confidences.
• Prompt composition: combine intent + persona + product prefs into agent prompts (1:1 mapping).
• Agent execution
  • Perceive via accessibility tree, maintain full episodic memory, use schema-constrained LLM outputs to select actions, record reasoning and outcomes, check termination.
  • Safety guardrails to avoid pathological runs.
• Ground-truth construction & confounder control
  • Filter theme transitions to exclude promotions, seasonality, assortment/pricing changes and validate with double machine learning; use DOM and screenshot inspection to characterize change magnitude.
• Evaluation & statistical methods
  • Bootstrapped resampling (50–700 agents; 1,000 repeats) to assess stability and select agent count.
  • Metrics: sign alignment (% shops where agent and human A2C change directions match), alignment probability (Bayesian posterior), Pearson correlation for magnitudes across buyer clusters.
  • LLM-based post-hoc log filtering to flag undesirable runs.

Implications for AI Economics

• Efficiency and experimentation economics
  • Platforms and merchants can reduce the direct cost and opportunity cost of live A/B testing: faster iteration (hours vs weeks), less real-user exposure to harmful variants, and lower traffic-split requirements. This can materially lower the marginal cost of running design/UX experiments and accelerate product-cycle speed.
  • Lower experimentation costs may increase the number and scope of UI and product experiments, potentially raising aggregate innovation rates in online retail interfaces and platform services.
• Risk management and welfare
  • SimGym enables safer pre-testing of risky or radical UI changes (which historically merchants avoid due to conversion risk). This could change merchant behavior toward bolder experimentation with possible efficiency gains but also risk of over-optimizing manipulative interfaces if incentives misalign.
  • By approximating heterogeneous treatment effects across buyer personas, platforms can better predict distributional impacts (which segments win/lose) and design more equitable rollouts.
• Econometric and causal-inference applications
  • If broadly validated, synthetic agents grounded in per-shop histories could complement randomized experiments by screening treatments and prioritizing those with predicted positive lift—improving the allocation of experimental traffic and lowering sample size needs.
  • The paper’s use of double machine learning to control confounders illustrates a hybrid approach: causal identification for ground-truth plus predictive simulation—this pattern could be extended to policy counterfactuals or pricing experiments.
• Limitations and external validity concerns
  • External validity: agents are trained from historical clickstreams and may fail to anticipate genuinely novel human responses to UI changes (especially visual/affective elements not captured in current accessibility-tree-based perception). This constrains how much economic inference one can draw from simulations alone.
  • Small ground-truth sample and filtering for low‑confounder cases mean current evidence is promising but preliminary; scaling validation is necessary before relying on simulations for high-stakes decisions.
  • Strategic behavior and gaming: if platforms or merchants rely on simulations for rollouts, there could be incentives to optimize for agent-predicted outcomes rather than human welfare, leading to second-order effects.
• Policy and market-structure effects
  • Lowering experimentation costs can advantage larger platforms that can run and validate extensive simulation pipelines, potentially increasing competitive asymmetries unless simulation tools become widely accessible.
  • Regulators and platform governance may need to consider standards for validating synthetic experiments when those outputs drive platform-wide policy or automated rollouts.

Summary judgment: SimGym offers a practical, well‑engineered step toward using LLM agents for offline A/B pre-testing in e-commerce. The approach combines per-shop persona grounding, live-browser LLM agents with episodic memory, and careful ground-truth construction. Preliminary validation shows meaningful predictive power (directional alignment and ~0.65 correlation at scale), but broader validation across more shops, inclusion of vision-based changes, and attention to external validity and incentive effects are necessary before substituting simulated for real randomized experiments in high-stakes contexts.