An LLM-powered 'A/B Agent' automates strategy iteration for industrial recommender A/B testing, delivering a reported 4.829% uplift in Cart GMV in a live short-video e-commerce system while maintaining guardrail metrics; however, the paper provides limited statistical details on the production A/B design and robustness checks.

Main Finding

A/B Agent is a closed-loop, agentic system that automates industrial A/B strategy iteration by structuring historical experiments into a hierarchical “experience tree,” retrieving context-matched evidence via a multi-path Tree-RAG pipeline, generating executable strategies, and autonomously tuning them using online A/B feedback. In real-world deployment on a short-video e‑commerce recommendation system it produced a 4.829% lift in Cart GMV while preserving guardrail metrics, and outperformed several LLM- and RAG-based baselines in offline evaluations.

Key Points

• Problem addressed

  • Industrial recommendation strategy tuning is labor-intensive, fragmented across heterogeneous records, and requires repeated manual design → long iteration cycles.
  • Flat RAG retrieval misses hierarchical context (domain, scenario, pipeline stage, objective), limiting transfer and continuous refinement.

• Core contributions

• Hierarchical strategy experience tree: decomposes historical A/B records into atomic strategy chunks (motivation, mechanism, parameters, applicability, outcomes, risks) and organizes them by domain → scenario → recommendation stage → objective to support precise retrieval and transfer.
• Tree-RAG multi-path retrieval and reranking: combines sparse (exact terms) and dense (semantic) retrieval, then boosts candidates using tree-path relevance to prefer structurally matched evidence; Qwen-based reranker performs fine-grained scoring on applicability and constraints.
• Evidence-grounded strategy generation + judging: generates candidate, executable strategies adapted to available inputs, service costs, and valid parameter ranges; safety/feasibility checks are applied before launch.
• Experiment-guided self-evolution: organizes successive online variants into an experiment tree (parent–child, sibling comparisons), performs iterative local parameter search or mechanism replacement using A/B outcomes, and writes validated experiences back to the experience tree to form a self-improving loop.

• Empirical validation: offline evaluations across three industrial e‑commerce scenarios and an online deployment with a tangible GMV improvement.

• Quantitative results (selected)

  • Real-world deployment: +4.829% Cart GMV while maintaining positive guardrail outcomes.
  • Offline composite score: A/B Agent achieved average overall score = 7.244; beat Claude-Sonnet-4.6 by 1.3%.
  • Compared to strongest RAG baselines per scenario: improvements of 25.0%, 31.7%, and 23.5% in overall score.
  • Also reported consistent wins over GPT-5.5 on correctness and Claude-Sonnet-4.6 on novelty.

• Design/algorithms (high level)

  • Retrieval scoring: S_ret(c,q) = sum_m α_m Norm(S_m(c,q)) combining M retrieval paths (sparse/dense).
  • Tree-path relevance boost: B_tree(c,q) = sum_l β_l exp(−γ_l d_T(v^l_c, v^l_q)), where d_T is shortest-path distance in experience tree; used to prefer structurally matched candidates.
  • Reranker and strategy generator adapt mechanisms to constraints (inputs, cost, parameter ranges) and produce multiple candidate setups for experimentation.
  • Experiment-tree comparisons guide parameter weighting choices and next-round A/B tests.

Data & Methods

• Data

  • Constructed an industrial A/B dataset containing: structured strategy configurations, experimental outcomes, multi-dimensional metrics (core and guardrail), and deployment metadata.
  • Real-world deployment data: short-video e-commerce recommendation experiments measuring Cart GMV and several guardrail metrics.

• Methods / Pipeline

• Historical Experiment Structuring: normalize records to a schema (background, scenario, stage, objective, strategy description, parameter updates, metrics, risks, deployment conditions).
• Agentic Chunk Extraction: decompose records into atomic strategy chunks with multiple perspectives (problem, scenario, mechanism); apply schema and metric checks.
• Hierarchical Experience Tree: index chunks under semantic paths [domain, scenario, stage, objective]; allow multi-attach for transferable strategies.
• Multi-Path Retrieval: combine BM25/TF-IDF sparse retrieval and embedding-based dense retrieval; aggregate via weighted normalization.
• Tree-based Boost + Reranking: boost candidates structurally closer in the tree; Qwen-based reranker assesses applicability and constraints.
• Strategy Generation & Judging: produce executable candidate strategies (parameters within valid ranges); run safety/feasibility checks and optional human expert evaluation.

• Autonomous Experimentation Loop: launch A/B tests; monitor core & guardrail metrics; store variants into experiment tree; use parent/child/sibling comparisons to identify successful edits and guide parameter search; validated results update the experience tree.

• Evaluation

  • Offline human/expert scoring across three industrial scenarios vs. LLM and RAG baselines (metrics: overall score, correctness, novelty).
  • Online A/B deployment in production recommender system measuring GMV and guardrails.

Implications for AI Economics

• Productivity and labor effects

  • Automation of strategy design and A/B tuning can substantially reduce engineers’ time per experiment (hours per iteration), increasing experiment throughput and decreasing marginal cost of experimentation.
  • This is an instance of task automation that augments experienced engineers: the agent initializes strategies and performs routine tuning while experts can focus on higher-level decisions or edge cases.

• Returns to data and experimentation history

  • Firms with richer, well-structured experiment histories gain greater returns from such agents because the experience tree is the primary capital for transfer—amplifying returns to past experimentation investment.
  • This reinforces data- and experimentation-driven advantage, potentially increasing incumbent advantage and entry barriers for smaller firms lacking similar histories.

• Speed of innovation and welfare

  • Faster, cheaper experimentation can accelerate product improvement and personalization, increasing short-run consumer surplus via better recommendations and higher GMV for merchants/platforms.
  • However, the same tooling could be used to optimize for engagement/monetization in ways that harm long-term welfare (e.g., attention exploitation) unless guardrails are robustly enforced.

• Allocation of experimental resources & risk externalities

  • Autonomous agents may concentrate experiments on high-return segments (rich users/traffic), potentially widening heterogeneity in experience across user groups.
  • The closed-loop learning could propagate miscalibrated or biased strategies across scenarios if historical data encode biases; hierarchical transfer reduces mismatch but does not eliminate systemic error—necessitates monitoring and human oversight.

• Market structure and competition

  • Because the method increases the productivity of A/B-driven optimization, platforms that adopt it could see measurable revenue gains (e.g., the reported 4.829% GMV uplift). Widespread adoption could raise the industry baseline and intensify competition around experimentation infrastructure.
  • Firms that open-source or share structured experiment primitives might change competitive dynamics; otherwise, proprietary experience trees become strategic assets.

• Policy and governance

  • The system’s reliance on historical outcomes means governance should focus on (a) auditing experience-tree contents, (b) verifying guardrail constraints and multi-metric objectives, and (c) preventing amplification of undesirable behaviors (e.g., manipulative ranking changes).
  • Regulators and platform managers should ensure transparency on objectives and constraints used by automated tuning agents, and require human-in-the-loop checkpoints for high-risk changes.

• Economic value of structured knowledge engineering

  • The experience tree formalizes “organizational capital” from experiments; investing in structured, reusable representations of past A/B tests yields ongoing returns via transfer and faster iteration—an economic justification for deliberate knowledge engineering.

Limitations & caveats (economic perspective) - Effectiveness depends on the quality and coverage of historical data; small firms or new products may see limited benefit. - The approach can transfer interventions across contexts only to the degree that applicability checks and reranking avoid mismatches—misapplied transfers could generate negative externalities. - Aggregate welfare impact depends on how platform incentives and guardrails are specified; revenue-maximizing agents without robust constraints could impose hidden costs on users.

Summary A/B Agent operationalizes transfer learning over historical A/B experiments and closes the loop between strategy generation and online feedback. Economically, it lowers the cost of experimentation, raises returns to prior experimentation investment, and can materially increase firm-level metrics (GMV), but it also concentrates value in firms with rich experiment histories and poses governance and welfare questions that merit policy attention.