An LLM-augmented bandit harness automates recommender engineering, producing stable offline gains across benchmarks and delivering a small but measurable business uplift in a 7-day online A/B test (+2.084% ADVV, +0.534% Revenue, +0.559% Exposure).

Main Finding

RecHarness is a two-part automated optimization harness for recommender systems that combines a bandit-based router (Thompson sampling over interpretable "edit arms") with an LLM-driven code-generation module (conditioned on a compact Experiment Skill). By separating arm selection (scalar, posterior-driven) from hypothesis/code generation (textual reasoning), and by including a basin-aware jump/retuning mechanism, RecHarness delivers more stable, trial-efficient improvements than LLM-reasoning-only search. In offline benchmarks across sequential-recommendation and watch-time tasks and in a 7-day online A/B test on a large short-video ad platform, it produced measurable gains (online: ADVV +2.084%, Revenue +0.534%, Exposure +0.559%).

Key Points

• Two-level separation:
  • Bandit router: decides which semantic edit directions (arms) to try using scalar validation feedback and Thompson sampling over Beta posteriors.
  • LLM generator: given a selected arm, incumbent, execution logs, and Experiment Skill, generates concrete hypotheses and executable code edits.
• Arms are semantic edit dimensions (e.g., LR schedule, dropout, embedding size, loss function, pooling strategy). Arms are partitioned into Alocal (incremental) and Ajump (structural).
• Reward / posterior update:
  • Each trial yields a binary success rt(a) ∈ {0,1}, defined by validity, whether the candidate outperforms the group average, and whether it beats the incumbent.
  • Posteriors αa, βa updated accordingly; Thompson sampling samples ˜θa to select next arms.
• Experiment Skill:
  • A compact textual memory summarizing incumbents, recent successful edits, failure modes and logs; conditions the LLM to reuse lessons and avoid known bad edits.
  • Crucially, Experiment Skill informs the LLM but does not alter bandit posteriors (keeps scalar and textual evidence separate).
• Basin-aware jump + retuning:
  • Measure recent improvement rate; when stagnation detected, allow Ajump arms.
  • Structural jump candidates undergo local retuning (up to R rounds) and are accepted only if retuned branch outperforms incumbent by δjump, allowing structural changes time to adapt.
• Grouped parallel trials: can evaluate multiple arms concurrently under the same incumbent and normalize improvements within a group to compare fairly.
• Practical outcomes: more stable improvements under limited trial budgets compared to unconstrained LLM search; validated with both offline datasets and a production A/B test. Code released: https://github.com/6lyc/RecHarness.

Data & Methods

• Tasks / Datasets:
  • Sequential recommendation: four Amazon review subsets — Movies, Scientific, Electronics, CDs (each user/item ≥5 interactions; leave-last-out evaluation).
  • Watch-time / ranking: KuaiRec (dense user-video interactions).
• Metrics:
  • Sequential rec: HR@10, HR@20, NDCG@10/20; reports AvgHR@10 across Amazon datasets.
  • KuaiRec: WT-XAUC / WT-MAE (watch-time), WR-XAUC / WR-MAE (watch-ratio).
• Search protocol & budget:
  • Incumbent-based sequential trials: start from a cold-start template, one baseline trial, then iteratively generate candidate edits under selected arms until budget exhausted.
  • Each candidate is trained and validated in an isolated workspace; only executable candidates that improve the incumbent are promoted.
• Bandit formulation:
  • Arm set A = {a1,...,aK}; arms split into Alocal and Ajump.
  • For each arm a, maintain Beta(αa, βa). At round t sample ˜θa ∼ Beta(αa, βa); select top-G arms for parallel trials.
  • Normalize per-group improvements bAt(a) and compute binary success rt(a) based on candidate validity, relative group improvement, and beating incumbent.
  • Update αa ← αa + rt(a), βa ← βa + 1 − rt(a).
• Jump mechanism:
  • Track recent improvement rate over window W; if below threshold τ, activate jump arms.
  • Evaluate jump candidates with up to R retuning rounds; accept if max retuned validation − incumbent > δjump.
• Experimental evaluation:
  • Offline comparisons across tasks, datasets, and model backbones vs. LLM-reasoning search baselines.
  • Online A/B test: 7-day deployment on large-scale short-video advertising platform showing business metric gains reported above.

Implications for AI Economics

• R&D productivity and cost-efficiency:
  • RecHarness formalizes trial allocation under budget constraints, increasing the return on expensive training-validation trials. This can reduce engineering hours and compute wasted on low-yield hypotheses, lowering marginal costs of model improvement.
• Labor and skill composition:
  • By automating iterative engineering decisions (direction selection + hypothesis generation), RecHarness can shift human engineers toward higher-level strategy, arm-definition, and safety/governance roles, potentially changing demand for routine model-tuning labor.
• Competitive dynamics & time-to-market:
  • Faster, more reliable iteration reduces time-to-deploy improvements. Platforms that adopt such harnesses can achieve incremental business gains more frequently, reinforcing winner-take-most dynamics in attention-driven markets.
• CapEx vs OpEx trade-offs:
  • The system concentrates experiments into higher-value arms and avoids fruitless explorations; firms can reallocate budget from brute-force search (CapEx-intensive compute) to targeted engineering/validation (OpEx), improving ROI on modeling pipelines.
• Market entry & barriers:
  • Lower experimentation cost may reduce barriers for smaller firms to iterate model improvements, potentially increasing competition. Conversely, the need for production-scale isolated trials could keep initial barriers.
• Risk management and governance:
  • Automated model edits in production raise governance concerns: detecting distributional drift, ensuring fairness and safety of automated changes, and auditing LLM-generated code. Economic value from automation must be balanced with operational risk and oversight costs.
• Product metric externalities:
  • Small percentage gains in core metrics (ADVV, Revenue, Exposure) can compound platform revenues. The marginal economic value of automated optimization depends on scale—minor improvements can translate to substantial monetary impact at large user bases.
• Future markets & services:
  • This architecture suggests new services: automated experiment orchestration as a SaaS, marketplaces for curated arm sets or distilled Experiment Skills, and tooling for auditing/insurance of automated model evolution.

Overall, RecHarness demonstrates a pragmatic design pattern for budget-constrained, automated ML engineering in recommender economics: combine principled exploration (bandits) for where to search with LLM-enabled exploitation for how to implement edits, plus structural mechanisms (retuning) to escape local optima—yielding measurable business value while reshaping the costs and organization of model development.