A hybrid conversational engine at Baidu decouples instant replies from asynchronous tool-driven work, preserving conversational flow while integrating delayed results; in production this architecture reduced latency and—by observational metrics—tripled Day-7 retention and raised complex task completion to 65%.

Main Finding

DuCCAE is a production hybrid architecture (deployed within Baidu Search) that reconciles real-time conversational responsiveness with reliable, long-horizon agentic execution by decoupling a fast, low-latency interaction path from an asynchronous multi-agent execution path and synchronizing them via a shared session state. The system improves agentic execution reliability and dialogue quality while meeting strict real-time latency budgets, and—in production since June 2025—produced strong business outcomes (Day‑7 retention tripled to 34.2% and Complex Task Completion Rate rose to 65.2%).

Key Points

• Architectural innovation
  • Latency‑Decoupled / Dual‑Track design: a Fast Track for sub-second persona‑consistent replies and a Slow Track for asynchronous planning, tool calls and multi‑agent execution.
  • Event‑driven synchronization through a Unified Shared State that preserves session context and execution traces, enabling asynchronous results to be reintegrated into the live dialogue without breaking immersion.
  • Five tightly integrated subsystems: Info (multimodal perception & memory), Conversation (intent stratification & low‑latency rendering), Collaboration (team dispatcher, planner, executor, generator), Augmentation (tools/retrieval/protocols), Evolution (episode judging, automated post‑training).
• Runtime behavior and routing
  • Query Understanding stratifies intents into three tiers (Tier‑1 lightweight; Tier‑2 deterministic tool calls; Tier‑3 complex domain requests) and routes them accordingly.
  • Fast Track guarantees Time‑to‑First‑Token (TTFT) under ~500 ms for simple turns; Slow Track handles heavy‑tail tasks asynchronously.
  • Planner decomposes tasks into serial and parallel sub‑tasks to optimize latency; intermediate results are written back to shared Task Context for continuity.
• Perception and state
  • Info System converts audio/video into unified textual context via streaming ASR and a lightweight VLM (captions/semantic cues) to avoid heavy VLM latency.
  • Dual memory: User Memory (persistent personalization) vs Agent Memory (execution state & system-of-thought) to avoid contaminating long‑term user profile with ephemeral task traces.
• Continuous improvement
  • Evolution system collects interaction episodes, judges outputs (LLM + human judges), filters episodes and drives SFT / RL‑FT updates to models and agents.
• Empirical/production outcomes
  • Offline and online evaluation (Du‑Interact dataset + large‑scale production metrics) show improvements over strong baselines in dispatch precision, task success, dialogue quality, and latency compliance.
  • Production KPIs reported: 3× Day‑7 retention (34.2%), Complex Task Completion Rate 65.2%, serving millions of users since June 2025.

Data & Methods

• Datasets and benchmarks
  • Du‑Interact: an internal dataset used for offline benchmarking of multi‑intent interaction, dispatch precision and dialogue quality (paper reports offline evaluation but full dataset details not provided in excerpt).
  • Large‑scale production evaluation inside Baidu Search: A/B / post‑launch metrics and user behavior analyses (exact experimental design not fully enumerated in excerpt).
• Metrics reported
  • Latency: TTFT target for Fast Track <500 ms.
  • Quality: dispatch precision, task success / complex task completion.
  • Business: Day‑7 user retention, task completion rates.
• System components & algorithms
  • Query Understanding: intent complexity classifier → three‑tier routing policy.
  • Collaboration Plan–Execute–Generate pipeline:
    • Team Dispatcher selects specialized subagents via semantic matching of intent embeddings to agent profiles.
    • Planner builds dependency graphs and parallelizes non‑blocking work.
    • Executor runs tools/subagents and writes intermediate results to shared Task Context.
    • Generator synthesizes final user deliverables (LLM or VLM output).
  • Info pipelines: streaming ASR, lightweight VLM captioning, context fusion into text for LLMs.
  • Augmentation: external tool layer (search, APIs, media generation), retrieval augmentation.
  • Evolution: judge components (automated LLM judge + human judge), filtered episodes used for SFT/RL‑FT.
• Evaluation strategy
  • Combined offline benchmark comparisons (to zero‑shot / larger parameter baselines) and large‑scale production measurements to show improvements in both technical and user‑facing KPIs.
  • Emphasis on both system reliability (agentic execution) and interaction quality/latency.

Implications for AI Economics

• User retention and lifetime value
  • Improving conversational continuity and complex task reliability materially increases retention (example: 3× Day‑7 retention). For product owners, higher retention raises user lifetime value (LTV) and monetization potential of conversational interfaces.
• Latency vs capability tradeoffs and cost structure
  • The dual‑track decoupling is a cost‑effective design: it avoids scaling heavy models to strict real‑time budgets (reducing peak real‑time compute needs) while supporting expensive background work. This architecture can reduce perceived latency without proportionally increasing real‑time compute costs.
  • However, asynchronous background execution and specialized subagents increase total compute and engineering costs (more services, orchestration, storage for shared state, continual training). Economic analysis must compare marginal revenue gains (retention, conversion) to these added OPEX/CAPEX.
• Economies of specialization and modularity
  • The Team Dispatcher + specialized agents enables reusing small, domain‑tuned models (economies of specialization), which can be cheaper and more reliable than monolithic giant models for certain tasks. Modularity also supports differential pricing or tiered offerings (basic instant chat vs paid deep planning).
• Platform effects and lock‑in
  • A shared session state, rich personalization memory, and continuous evolution pipeline create switching costs and data lock‑in (valuable training data from interactions). This can strengthen incumbent platforms’ competitive moats but raises concerns about market concentration.
• Measurement & incentives
  • The Evolution loop turns interactions into labeled episodes via automated judging and selective human oversight, producing training data that amplifies system strengths. Economically, this is a feedback loop where product usage funds future quality improvements—positive reinforcement for market leaders.
• Labor and task substitution effects
  • High success rates on complex tasks (65.2% Complex Task Completion) imply substitution of human labor (e.g., travel planning, customer support). That affects labor demands in service sectors and may shift firm cost structures (fewer human agents, more engineering).
• Quality externalities and trust
  • Preserving persona consistency and avoiding “dead air” sustains trust—important because trust influences adoption and monetization. Failures in asynchronous integration (mistimed or incorrect updates) could produce reputational risk and regulatory scrutiny.
• Regulatory and privacy considerations
  • Continuous memory, personalization and large‑scale logging of multi‑modal sessions raise data governance, privacy and consent costs. Compliance (e.g., data deletion, opt‑out) imposes additional operational burdens that affect economics.
• Strategic product design implications
  • Firms can differentiate by engineering integration (orchestration, orchestration + evolution) rather than only model size, shifting competition towards system engineering capabilities.
  • Hybrid architectures enable offering differentiated SLAs (guaranteed sub‑second responsiveness for basic interactions; background premium capabilities), enabling new pricing or subscription strategies.

Summary takeaway for economists and product managers: DuCCAE illustrates that architectural design (latency decoupling, specialized agents, shared state and continuous evolution) can unlock substantial user and task‑level value without relying solely on scaling monolithic LLMs; but these gains come with tradeoffs in background compute, engineering complexity, data governance costs, and competitive dynamics that must be explicitly modeled when evaluating ROI and platform strategy.