A proactive LLM assistant, Vigil, embeds into on-call customer–analyst dialogues on ByteDance's cloud platform and assists human support throughout the entire case lifecycle rather than only before escalation. Deployed for over ten months, Vigil also auto-extracts lessons from human-resolved cases to update its capabilities, demonstrating practical operational gains in a large-scale production environment.

Main Finding

Vigil is a deployed proactive LLM-powered agent that stays active after escalation to human analysts in large-scale cloud on-call dialogues. By (1) continuously monitoring analyst–customer conversations and proactively answering in-scope sub-questions, and (2) autonomously extracting and adding verified resolution knowledge from those conversations into its retrieval corpus, Vigil reduces analyst workload and improves end-to-end support efficiency without expensive model fine-tuning. The system was deployed in production on ByteDance’s Volcano Engine for over ten months, with quantitative evaluation and case studies showing practical operational gains. The open-source implementation is available at https://github.com/volcengine/veaiops.

Key Points

• Problem addressed
  • Reactive LLM agents typically disengage after escalation to humans, missing opportunities to answer later sub-questions that are within the agent’s capabilities.
  • Static, manually-maintained knowledge bases lag behind dynamic, fast-evolving on-call knowledge (temporary workarounds, new symptoms), limiting agent usefulness.
• System design (Vigil)
  • Proactive operation: Vigil monitors ongoing on-call chat between customers and human analysts and intervenes without explicit invocation when it detects an in-scope question.
  • Scope-aware question identification: New customer messages are classified (Within Scope / Out of Scope / No assistance needed) using LLM-based prompting to decide whether to intervene.
  • Question rewriting: Multi-turn dialogue context is used to convert implicit sub-questions into explicit, self-contained queries before retrieval and generation.
  • Answer generation via RAG: Multi-path retrieval aggregates domain documents and historical QA pairs, reranks results, and supplies grounded context to the LLM answerer. Vigil can also call tools (e.g., logs, alerts, diagnostic metadata) for grounded responses.
  • Answer deduplication: Embedding-based semantic similarity is used to suppress redundant answers within a session; an empirical cosine similarity threshold (~0.7) is used to decide redundancy.
  • Proactive response card UI: Visible, clearly labeled cards with citations and an Accept button separate agent suggestions from human messages and collect verified feedback.
• Continuous self-improvement
  • Vigil ingests outcomes of answered/unanswered questions and analyst/custome r follow-ups to extract canonical Q–A pairs and artifacts, then keeps/updates/deletes items in its knowledge collection.
  • The mechanism is non-parametric (updates retrieval corpus) rather than relying on costly offline fine-tuning, enabling faster incorporation of new operational knowledge.
  • The Accept interaction from the UI serves as high-quality ground truth for learning.
• Deployment and evaluation
  • Deployed on Volcano Engine (ByteDance) for 10+ months; authors report quantitative metrics and case studies demonstrating improved operational efficiency and on-call quality (details and numbers are presented in the full paper).
  • Designed to complement, not replace, first-line reactive agents and human analysts.

Data & Methods

• Operating context and data
  • Real-world production environment: Volcano Engine cloud platform, which handles thousands of support tickets per day.
  • Data sources used by Vigil: live on-call dialogues, historical on-call Q–A pairs, domain document knowledge base (product/service docs), system artifacts (alerts, logs).
• Core modules and algorithms
  • Identification/classification: LLM prompt-based classifier to decide whether a new message is within the agent’s scope.
  • Question rewriting: Contextual rewriting to produce explicit questions for retrieval (leveraging standard LLM techniques for coreference resolution and ellipsis handling).
  • Retrieval-Augmented Generation (RAG): Multi-path retrieval (domain docs + historical QA), reranking model to prioritize relevant documents, then LLM synthesis of answer with citations.
  • Tools integration: Ability to invoke system diagnostic tools (logs/alerts) to ground answers in live system state.
  • Deduplication: Pretrained sentence embeddings and cosine similarity threshold (~0.7) to detect and suppress redundant answers within a session.
  • Self-improvement pipeline: After-session review that extracts high-confidence answers (using Accept feedback and analyst confirmations), deduplicates, and updates the searchable knowledge store; supports keep/update/delete logic for entries.
• Evaluation approach
  • Production deployment for longitudinal evaluation (10+ months).
  • Quantitative metrics (reported in the paper): measures of operational efficiency and support quality — e.g., reductions in analyst message load, response latency, or similar KPIs (specific metric values are in the full text).
  • Case studies illustrating behavior during incidents (e.g., temporary workaround propagation, reduced repeated explanations).

Implications for AI Economics

• Labor productivity and cost
  • Augmentation: Vigil reduces repetitive work by proactively answering in-scope follow-ups, increasing human analyst throughput and reducing the marginal labor required per ticket.
  • Cost savings: Fewer analyst messages and shorter resolution times imply lower operational costs per ticket; non-parametric updates reduce recurring engineering/maintenance costs versus frequent finetuning.
  • Labor composition effects: Proactive agents may shift analyst tasks toward higher-complexity diagnostics and oversight, potentially increasing demand for specialized skill while reducing demand for routine support roles.
• Value capture and pricing
  • Improved SLAs: Faster incident resolution and lower wait times can reduce downtime costs for customers; providers could adjust pricing or offer differentiated support tiers that monetize improved responsiveness.
  • Customer retention and platform competitiveness: Better support experience is an intangible asset that can increase customer retention and LTV (lifetime value).
• Knowledge externalities and public goods
  • Rapidly shared, automatically propagated fixes (via the self-improvement loop) create positive externalities across users experiencing similar failures; this increases the marginal value of each resolved incident.
  • However, there are risks if incorrect agent-propagated fixes spread (negative externalities), implying the need for verification feedback (the Accept button and analyst confirmation mitigate this).
• Operational risk, governance, and measurement
  • Risk of error propagation: Learning from interactions risks incorporating incorrect/unsafe fixes into the KB; governance mechanisms and human-in-the-loop verification are economically important to prevent costly misresolution.
  • Measurement challenges: Properly quantifying Vigil’s economic impact requires careful causal evaluation (e.g., randomized rollouts, counterfactuals) to separate agent effects from temporal variation in incident types or staffing.
  • Long-run dynamics: Agents that internalize recurring knowledge reduce marginal information rents for human experts, which could depress wages for routine support but increase the premium on high-end problem-solving skills.
• Research directions for AI economics
  • Quantify labor substitution vs. complementarity across task types and over time as the knowledge base grows.
  • Estimate cost–benefit of non-parametric continuous learning vs. periodic model retraining (engineering and data-maintenance costs).
  • Study welfare effects of faster incident propagation of workarounds (both beneficial and harmful externalities).
  • Analyze optimal pricing strategies for cloud providers that deploy proactive support agents (e.g., tiered SLAs, premium instant-response options).
  • Investigate the effect of proactive agents on staffing models, training investments, and human capital accumulation in technical support organizations.

Limitations and caveats (from an economic lens) - The paper reports deployment and positive outcomes, but precise effect sizes and experimental causal identification are necessary to make firm economic conclusions. - There are governance costs (verification, monitoring, legal/compliance) that offset some operational savings; these should be quantified. - Potential long-term impacts on worker skills and labor markets need longitudinal study.

If you want, I can: - Extract specific quantitative results and KPIs from the full paper (if you provide the remaining evaluation sections), or - Draft an experimental design for measuring Vigil’s causal impact on labor costs and response times in a cloud-support firm.