Meituan’s ATLAS framework diagnoses where multi-step AI assistants fail during execution and across subsequent turns, then uses calibrated diagnostic signals to optimize policies—live A/B tests on Meituan Xiaotuan report improved user engagement and downstream business outcomes.
ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents
ATLAS is a dual-horizon diagnostic evaluation framework that generates calibrated trajectory-wise and user-wise signals to localize and measure failures of industrial tool-use agents, and its deployment on Meituan Xiaotuan yielded offline diagnostic fidelity and online A/B improvements in engagement and business metrics.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Large language model (LLM) agents are increasingly deployed in user-facing services that require iterative tool use under dynamic business conditions. Reliable evaluation is essential for sustained improvement: it must reveal capability deficiencies, inform priorities, and assess interventions. Yet industrial agent service unfolds both through the iterative trajectory of a current request and through continued user interaction. Final-outcome assessment can therefore obscure where deficiencies arise and whether later service remains aligned with context from earlier exchanges. We propose ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. At the request horizon, trajectory-wise diagnostic signals relate deficiencies to execution locations and capability concerns. At the interaction horizon, user-wise signals assess whether service remains responsive across continued interaction. Together, these views provide structured diagnostic evidence for analyzing execution deficiencies and sustained service behavior. ATLAS instantiates them as executable signals with explicit evidence scopes and decision boundaries. LLM judge interfaces are calibrated against high-confidence references from real business logs; when needed, their decision behavior is distilled into efficient diagnostic models for lower-latency, lower-cost evaluation. The resulting feedback supports policy optimization. We evaluate ATLAS on Meituan Xiaotuan production traffic. Offline experiments assess diagnostic-signal fidelity and replay-based policy improvement, while online A/B experiments show concurrent gains in user engagement, downstream business outcomes, and sampled human-audit quality.
Summary
Main Finding
ATLAS (Ability Taxonomy and Logical Assessment System) is a dual-horizon diagnostic evaluation framework for industrial tool-use LLM agents. It provides trajectory-wise (within-turn) and user-wise (across-turn) diagnostic signals with explicit evidence scopes and decision boundaries, calibrates LLM-based judges on high-confidence business logs, and can distill signals into efficient diagnostic models. Applied to Meituan Xiaotuan production traffic, ATLAS both guided policy optimization and produced measurable gains in online A/B tests (user engagement, downstream business outcomes, and sampled human-audit quality).
Key Points
- Dual-horizon view
- Within-turn (trajectory-wise): preserves the multi-step execution trace for a single request to localize where failures arise.
- Across-turn (user-wise): tracks whether agent behavior remains responsive to evolving user context across lifecycles (context carry-forward, feedback incorporation, preference alignment, goal progress, user satisfaction).
- Structured diagnostic space (within-turn matrix)
- Execution locations: Thinking & Reflection, Tool & Skill Execution, Response Generation.
- Capability concerns: Relevance, Factuality, Timeliness, Reliability, Intent & Planning.
- Norms & Compliance as an independent guardrail (safety, privacy, format/structural validity).
- Fine-grained signals populate the intersections (e.g., Rewrite Validity, Supply Relevance, Tool-Call Hallucination, Outdated Supply Detection).
- Executable signals
- Each signal specifies evidence scope and decision boundary (how to decide pass/fail or graded score).
- Semantic signals use LLM judges calibrated on high-confidence reference sets from actual business logs.
- Selected signals distilled into lightweight diagnostic models for lower latency and cost at scale.
- Diagnosis-driven iteration
- Signals serve both as diagnostic reports and as multi-dimensional training feedback for policy optimization.
- The framework preserves semantic meaning of signals from evaluation through optimization and back to reassessment.
- Real-traffic validation
- Offline: diagnostic-signal fidelity checks and replay-based policy improvement experiments.
- Online: A/B experiments on Meituan Xiaotuan showing concurrent improvements in engagement, business outcomes, and human-audit quality.
Data & Methods
- Data source: production interaction logs and business data from Meituan Xiaotuan (real local-services traffic at scale).
- Signal construction:
- Define concrete diagnostic signal families tied to observable behavior (tool calls, intermediate reasoning artifacts, final responses) with explicit evidence scopes and decision thresholds.
- Create high-confidence reference sets sampled from real logs to serve as calibration targets for semantic judgments.
- Calibration:
- LLM judge interfaces are calibrated against these human-validated reference sets to align decision behavior with business-grounded judgments.
- Efficient implementation:
- Distill calibrated LLM-based judges into compact diagnostic models for throughput-conscious evaluation pipelines.
- Experiments:
- Offline fidelity evaluation to measure judge/model agreement with references.
- Replay-based policy optimization using multi-dimensional diagnostic feedback (turn-level and user-level signals).
- Online A/B testing in production to measure effects on user engagement, downstream business KPIs, and sampled human-audited quality.
- Example failure modes analyzed include intent-altering query rewrites, tool-call hallucinations, irrelevant supply retrieval, and outdated supply detection.
Implications for AI Economics
- More efficient allocation of development resources
- Fine-grained, interpretable diagnostic signals let teams prioritize fixes that have the highest expected impact on product/business objectives rather than optimizing undifferentiated end-to-end metrics.
- Better ROI measurement for agent improvements
- By linking diagnosis to downstream business outcomes (measured in A/B tests), organizations can quantify the economic return of specific capability investments.
- Lower evaluation cost and faster iteration
- Calibration + distilled diagnostic models reduce per-query evaluation cost and latency, enabling higher-frequency experimentation and shorter iteration cycles.
- Improved decision-making under dynamic externalities
- Signals for timeliness, supply relevance, and reliability help internalize the economic impact of real-world state changes (e.g., inventory, availability), reducing the risk of economically costly mistaken recommendations.
- Product-market competition and platform advantages
- Platforms that adopt rigorous diagnostic evaluation can iterate faster on high-value capabilities (e.g., context retention, tool reliability), potentially capturing more user engagement and revenue.
- Mitigation of externalities and regulatory risk
- Norms & Compliance guardrails embedded across evaluation help quantify and control safety/privacy risk, which has economic value by lowering compliance costs and reputational exposure.
- Costs and risks to consider
- Building calibrated references and human audits requires upfront labeling investment; signals can be gamed if used naively as optimization targets; ensuring alignment between diagnostic signals and business KPIs is essential to avoid perverse incentives.
- Generalizability
- The ATLAS structure is reusable across industrial tool-use domains (local services, recommendation, finance, route planning), enabling more comparable economic evaluations of agent deployments across sectors.
Overall, ATLAS operationalizes diagnostic evaluation so that engineering, product, and economics teams can translate observable agent deficiencies into prioritized, measurable, and cost-aware improvement actions.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper validates the evaluation framework on production traffic and reports online A/B improvements in engagement and business outcomes, which is strong practical evidence; however, the excerpt lacks detailed experimental design, randomization, sample sizes, effect magnitudes, statistical significance, and robustness checks necessary to judge causal claims as high-strength.
Methods Rigormedium — The framework is well-structured and operationalized (calibrated LLM judges, distilled diagnostic models, offline and online validation), but the provided text omits key methodological details (randomization protocol, sample sizes, pre-registration, metrics definitions, effect sizes, confidence intervals, and potential confound controls), limiting assessment of internal validity and reproducibility.
SampleProduction local-services interaction logs and real user requests from Meituan Xiaotuan (an industrial assistant operating at large scale), high-confidence reference sets derived from business logs for calibrating LLM judges, offline replay datasets for diagnostic fidelity and policy simulation, and online A/B traffic comparing ATLAS-optimized policies to baseline; human-audits sampled for quality checks.
Themeshuman_ai_collab adoption
IdentificationOnline A/B experiments on production Meituan Xiaotuan traffic (implied randomized split) comparing an ATLAS-optimized policy to baseline, supplemented by offline replay experiments and calibration of LLM-based evaluators against high-confidence reference sets from real business logs.
GeneralizabilityValidated in a single industrial domain (Meituan Xiaotuan local-services); results may not generalize to non-local-services domains or different product designs., Depends on access to extensive business logs and domain-specific reference sets; smaller organizations may lack similar data for calibration., Relies on domain-specific diagnostic signal definitions and LLM-judge calibration, which may not transfer across languages, cultures, or regulatory environments without retraining., Operational gains may depend on Meituan’s specific tooling, infrastructure, and user population, limiting external validity to other platforms or scales., Effectiveness may vary by agent architecture and the nature of external tools/skills available to the agent.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| ATLAS evaluates industrial tool-use agents at two complementary horizons: the request horizon, which analyzes a complete execution trajectory, and the interaction horizon, which assesses continued service across user interactions. Other | positive | Coverage and structure of diagnostic evaluation across individual requests and continued interactions |
Reading fidelity
high
Study strength
low
|
not reported
|
| Within-turn ATLAS signals localize capability deficiencies according to both where they arise in an execution and how the behavior falls short of service requirements. Decision Quality | positive | Localization and characterization of execution-level capability deficiencies |
Reading fidelity
high
Study strength
low
|
not reported
|
| Across-turn ATLAS signals assess whether user-relevant context, such as referents, intent, constraints, and corrections, is carried forward, updated, and used in subsequent service. Consumer Welfare | positive | Context consistency and responsiveness of service across multiple user turns |
Reading fidelity
high
Study strength
low
|
not reported
|
| ATLAS operationalizes its diagnostic structure as executable signals with explicit evidence scopes and decision boundaries. Organizational Efficiency | positive | Operationalizability and reproducibility of agent diagnostic evaluation |
Reading fidelity
high
Study strength
low
|
not reported
|
| LLM-based diagnostic signals in ATLAS are calibrated against high-confidence references constructed from real business logs. Ai Safety And Ethics | positive | Fidelity of semantic diagnostic signals |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Selected ATLAS diagnostic signals can be distilled into efficient diagnostic models to support lower-latency and lower-cost evaluation at scale. Organizational Efficiency | positive | Evaluation latency and evaluation cost |
Reading fidelity
high
Study strength
low
|
not reported
|
| ATLAS diagnostic feedback supports policy optimization by converting observed capability deficiencies into explicit, multidimensional improvement targets. Organizational Efficiency | positive | Policy improvement and optimization feedback quality |
Reading fidelity
high
Study strength
medium
|
not reported
|
| In online A/B experiments on real local-services interactions, an ATLAS-optimized policy improved user engagement, downstream business outcomes, and sampled human-audit quality. Consumer Welfare | positive | User engagement, downstream business outcomes, and sampled human-audit quality |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The paper validates ATLAS using production traffic from Meituan Xiaotuan through offline evaluation of diagnostic signals and diagnosis-driven optimization, together with online A/B experiments. Other | positive | Diagnostic-signal fidelity and policy improvement under offline replay and online deployment |
Reading fidelity
high
Study strength
medium
|
not reported
|