A recursive oversight system lets non-expert users steer LLMs to produce near-expert product requirements, yielding a 54% alignment gain on web-development tasks; the approach can be optimized through reinforcement learning using only online user feedback.

Main Finding

Scalable Interactive Oversight (SIO) — an agentic framework that decomposes complex user intent into a recursive tree of low-burden decisions and aggregates node-level preferences — substantially improves alignment between non-expert users and powerful LLM executors. In web-development (vibe coding) tests, SIO raises Product Requirements Document (PRD) alignment vs. non-interactive and vanilla-interaction baselines (e.g., average PRD alignment: GPT-5 0.503 → 0.670; Gemini-2.5-pro 0.359 → 0.554, a +54% relative gain). The framework also yields better full-stack implementations and can be optimized via reinforcement learning from online user feedback.

Key Points

• Problem framed as scalable oversight: non-expert users cannot fully specify or verify complex long-horizon tasks while models are strong executors. The Sandwich Protocol is used to evaluate how well weak human supervision can be amplified toward expert-level alignment.
• Core idea: decompose the global task into a tree of localized decisions. At each node the system asks closed-form, low-burden queries (selection, ranking, Don’tCare/Don’tKnow), collects node preferences, propagates and aggregates them, and adapts the remaining tree interactively.
• Mechanisms enabling scalability:
  • Simplifying supervision (comparative/selection judgments reduce cognitive load).
  • Amplifying weak signals via recursive preference propagation across nodes.
  • Arbitrary-depth tree structure to scale to long-horizon tasks.
• Empirical effects:
  • Substantial improvements in PRD alignment across models (examples: GPT-5 avg 0.503 → 0.670; Claude-sonnet-4.5 avg 0.565 → 0.618; Gemini-2.5-pro avg 0.359 → 0.554).
  • Largest gains in core-function module (Module 2) where user intent is most critical.
  • Alignment grows monotonically as more interaction nodes are processed.
  • Final website implementations also show gains (LLM-judge: 0.338 → 0.656; Human-judge: 0.453 → 0.520).
• Learning claim: the node-level interaction signals can serve as rewards for reinforcement learning, enabling the interaction policy itself to be optimized from online feedback.
• Robustness: includes mechanisms to handle ambiguous or uncertain user responses (clarification cycles) and to allow the agent to inject expertise when relevant (balancing preference and best practice).

Data & Methods

• Evaluation domain: full-stack website development; alignment measured at PRD level as a proxy for final implementation correctness.
• PRD construction: crawled production sites + search-augmented LLMs to assemble ground-truth intent PRDs structured into five modules: product overview, core function, non-functional requirements, business rules, user experience.
• Sandwich Protocol: three roles — non-expert (simulated), model (strong LLM generates PRD), expert (holds ground-truth intent I used to score alignment). Alignment score = fraction of atomic requirement rubrics satisfied.
• Test set: 37 synthesized cases (initial user queries) derived from real sites.
• User simulation: deepseek-R1 used to simulate non-expert users guided by the target intent I; validated against real interactions (details in appendix).
• LLMs used:
  • PRD generators (models to be aligned): GPT-5, claude-sonnet-4.5, gemini-2.5-pro.
  • Interaction agents typically used same model; for one baseline tree-update used o4-mini where necessary.
  • LLM-judge: Qwen3-235B-A22B-Instruct for large-scale automatic scoring; human judges also used for deployed implementations.
• Baselines: (1) Direct PRD generation via vibe-coding frameworks (codex, claude-code, gemini-cli), (2) Vanilla multi-turn free-form interaction.
• Metrics & analysis: module-level and overall alignment scores, alignment evolution over interaction nodes, ablations showing contributions of low-burden feedback and tree updating.
• Additional experiment: generated full implementations (claude-opus-4.5) from PRDs and evaluated implementations with LLM and human judges.

Implications for AI Economics

• Productivity & task automation:
  • SIO lowers the expertise threshold required to supervise complex automation, expanding the set of tasks that non-experts can safely delegate to LLMs. This can accelerate adoption of AI for sophisticated applications (e.g., software productization), raising aggregate productivity.
  • Because verification costs fall (smaller, local decisions instead of auditing large outputs), transaction and coordination costs in product development markets could decrease.
• Labor market effects:
  • Increased automation of expert tasks paired with human-in-the-loop oversight may shift demand away from pure implementation labor toward roles focused on oversight, interaction design, and exception handling. Some expert verification work may be substituted; other supervision/interaction-design tasks may be complementary and expand.
  • Lower barriers to entry for building software products could increase competition and crowd in smaller firms/individuals, affecting wages and market structure in software markets.
• Platform & business model impacts:
  • Platforms that provide optimized interaction policies (and can train them via RL from user feedback) may capture value by selling higher-alignment oversight as a productized service (e.g., premium “interaction policies” or oversight-as-a-service).
  • Monetization opportunities: charging for interaction depth, higher-quality oversight, or better RL-optimized agents that minimize user time per task.
• Incentives, measurement, and regulation:
  • Reliance on user feedback signals (often noisy and biased) as RL rewards raises incentive concerns: platforms may optimize for satisficing user convenience rather than long-run correctness or safety. Mechanisms to validate and debias feedback and to involve occasional expert audits will be economically important.
  • Regulatory scrutiny may focus on transparency of oversight policies and whether systems can reliably prevent mis-specification in high-stakes domains (finance, healthcare). Standards for auditability of the interaction tree and aggregated preferences may become market differentiators or regulatory requirements.
• Cost structure and capital intensity:
  • The SIO workflow adds interactive queries and state management overhead (compute + latency). Economic adoption will depend on trade-offs between increased alignment (reducing costly rework/errors) and added per-task compute/latency costs. For many business cases, reduced downstream failure costs will justify the overhead.
• Research and market opportunities:
  • Empirical work: quantify how SIO-like oversight changes adoption curves, product development cycle times, defect rates, and labor composition across firms.
  • Market for interaction policies: specialization and competition in optimizing agent questioning strategies (via RL) could form a new product category and source of rents for platform providers.
  • Welfare and distributional analysis: model how reduced verification costs and expanded automation affect income distribution between platform owners, skilled experts, and oversight workers.

Suggested economic research directions motivated by this paper: - Empirically measure cost savings and rework reduction from SIO vs. non-interactive workflows across firm sizes. - Model labor substitution/complementarity between experts and oversight/interactors, and simulate equilibrium wage responses. - Study incentive mechanisms and auditing regimes to ensure RL-optimized interaction policies do not drift toward gaming feedback signals. - Market design analysis for platforms that sell optimized oversight as a service, including pricing, competition, and regulatory constraints.