An LLM-driven delivery agent at Tencent sharply cuts data-warehouse delivery workload — median delivery time fell from 228 to 23 minutes and engineer effort from 95 to 11 minutes in production, with an 87% success rate and 73.5% autonomous submission rate.

Main Finding

SiriusDeliver is an end-to-end automation agent that substantially reduces human effort and time for data-warehouse (DW) task delivery while improving reliability. By combining a hierarchical delivery agent (scenario/context/artifact/platform skills), an artifact lifecycle control (pre- and post-execution diagnosis grounded in platform evidence), and trace-driven skill evolution (bounded updates from execution trajectories), Tencent deployed SiriusDeliver at scale and achieved large gains in success rate, automation, and engineer productivity.

Key Points

• System design
  • Delivery Automation Agent: maintains a structured delivery state and orchestrates reusable warehouse skills in four layers (scenario, context, artifact, platform) to produce an artifact bundle A = ⟨G (workflow), C (node artifacts, e.g., SQL/PySpark), P (task configs)⟩.
  • Artifact Lifecycle Control: two-stage verification — pre-execution diagnosis (deterministic platform evidence + LLM reasoning) to catch silent defects and performance risks before submission; post-execution diagnosis (log-grounded failure extraction + KB retrieval + root-cause revision) to produce traceable repairs.
  • Trace-driven Skill Evolution: groups delivery trajectories by outcome to generate bounded LLM-updates that (a) repair failures and (b) compress successful patterns into reusable skills; maintains memory of compressed trajectories, configs, and platform feedback.
  • Diagnosis items formalized: issue description, judgment conditions, required evidence, and execution workflow — prevents unsupported claims and makes repairs auditable.
• Empirical results (high level)
  • Offline (200 real-world delivery cases across 4 scenarios): end-to-end success increased by 14.5 percentage points (71.5% → 86.0% vs best baseline); token usage reduced by 30%; ablation studies confirm component contributions.
  • Production deployment (2 months on Tencent Cloud WeData): served 3,600 monthly active users across 6 teams, 18,240 delivery sessions; achieved 87.2% end-to-end success rate and 73.5% autonomous submission rate.
  • A/B test (1 month): median delivery time reduced from 228 to 23 minutes; engineer effort from 95 to 11 minutes; time-to-first-artifact from 44 to 2.6 minutes; manual intervention rate from 100% to 21%; final delivery success remained comparable.
• Practical observations motivating the design: majority of delivery time is not SQL coding but cross-platform context retrieval, dependency resolution, and coordination; one-shot LLM generation is brittle for multi-stage, dependency-rich DW deliveries.

Data & Methods

• Datasets and deployment
  • Offline evaluation: 200 real-world warehouse delivery cases, covering four delivery scenarios (e.g., synchronization, computation, workflow updates).
  • Online deployment: two-month roll-out on Tencent Cloud WeData across 6 business teams and 4 warehouse task types; recorded 18,240 sessions and 3,600+ MAUs.
  • One-month randomized A/B experiment comparing agent-driven workflows to human-driven baseline.
• Metrics
  • End-to-end delivery success rate (final correctness/completion of task submission and execution).
  • Autonomous submission rate (fraction of deliveries submitted without human intervention).
  • Time metrics: median end-to-end delivery time, engineer active effort time, time to first artifact.
  • Token consumption for LLM usage (efficiency metric).
• Methods & system evaluation
  • Comparative baselines: representative coding/agent baselines and skill-augmented agents.
  • Ablation studies: remove/disable components (e.g., lifecycle control, skill evolution) to measure their contribution.
  • Pre-execution checks: deterministic compilation/pre-compile checks, metadata verification, partition/predicate analysis to catch silent risks and performance hazards.
  • Post-execution pipeline: structured failure extraction from logs, retrieval of past fixes/troubleshooting docs, LLM-guided localized revisions.
  • Trace aggregation: grouping successful and failing trajectories to propose bounded, auditable updates to skills and memory.
• Quantitative highlights
  • Offline: +14.5 percentage points success, −30% LLM token consumption vs strongest baseline.
  • Online: 87.2% end-to-end success, 73.5% autonomous submission.
  • A/B: delivery time 228→23 min (≈90% reduction); engineer effort 95→11 min (≈88% reduction); manual intervention rate 100%→21%.

Implications for AI Economics

• Labor-productivity and cost implications
  • Large reductions in delivery time and hands-on engineer time imply significant labor cost savings per delivered warehouse task. For high-volume platforms (millions of tasks), these savings scale into substantial operational cost reductions.
  • Automation compresses low-value repetitive work (context retrieval, cross-platform coordination) into agent-managed workflows; human engineers shift toward higher-value tasks (design, validation, edge-case handling, governance).
• Demand-side shifts in labor markets
  • Decreased demand for routine DW delivery tasks; increased demand for roles in agent management, monitoring, data governance, and platform engineering to handle edge cases, skill evolution governance, and audits.
  • Skills premium likely for engineers who can work with and supervise automation agents, interpret artifact lifecycle reports, and design resilient data pipelines.
• Platform economics and vendor strategy
  • Cloud and DW platform providers gain incentives to integrate agent automation (differentiation, higher customer retention, and potential upsell of automated delivery tooling).
  • Economies of scale: skill libraries, diagnostic knowledge bases, and trajectory compression become more valuable as adoption grows — favoring larger platform players with broad user coverage.
• Risk, governance, and transaction costs
  • Adoption requires investment in auditability, traceable diagnosis, and bounded model updates to manage reliability and liability; these governance costs are non-trivial and affect net economic gains.
  • Potential negative externalities: automated but incorrect deliveries can create silent data-quality risks; vendors/operators must internalize costs of monitoring and corrective processes.
• Research and market opportunities
  • Cost–benefit and ROI studies: quantify direct labor savings, failure-repair costs, and required maintenance investments over time to guide enterprise adoption.
  • Market for “skill” marketplaces and certified diagnostic knowledge: third parties could sell vetted skill packs, repair rules, or trajectory templates.
  • Regulation and standards: need for standards around auditable automation, provenance of artifact revisions, and approval workflows — relevant to governance and compliance economics.
• Broader productivity vs. distributional effects
  • SiriusDeliver-style automation increases aggregate productivity in data engineering but will redistribute work composition and potentially reduce demand for entry-level delivery roles. Policy and firm-level retraining strategies will matter for managing these distributional impacts.

Takeaway: SiriusDeliver demonstrates that tightly integrated LLM-driven agents, when combined with deterministic platform grounding, lifecycle verification, and data-driven skill updates, can deliver substantial productivity and reliability gains in enterprise data-warehouse operations — producing clear economic benefits but also shifting labor demand and imposing governance and maintenance costs that must be managed for sustainable adoption.