An agentic LLM system that triangulates catalog, syndicated and identity-matched web evidence reliably enriches product attributes at scale—raising impression-weighted attribute coverage by ~90% in production—and a randomized A/B test shows surfacing those attributes on product pages increases checkout conversion by 0.48% (p=0.034).

Main Finding

TRACE is an agentic LLM framework that automates large-scale e-commerce catalog attribute enrichment by triangulating multi-source evidence (seller data, syndicated feeds, images, and identity-matched web search) with a two-agent verify-before-write architecture (ScoutAgent + JudgeAgent). In offline and production evaluations it achieves near-expert accuracy while scaling to millions of SKUs and improving business outcomes: human-validated extraction accuracy of 98.2% at 74.7% coverage on Grocery & Alcohol, judge-supported publication coverage up to ~85% across verticals, production enrichment of 31M SKUs (≈ +90% impression-weighted coverage), and an online A/B uplift in checkout conversion of +0.48% (power users +1.18%) and −1.08% missing/incorrect items.

Key Points

• Architecture
  • Two specialized agents:
    • ScoutAgent: collects and reconciles evidence from seller catalog fields, syndicated data, images, and agentic identity-matched web search; emits candidate attribute records with provenance and confidence.
    • JudgeAgent: independently verifies each candidate under a stricter evidence policy, may perform additional searches, and issues one of four verdicts: PASS, FAIL, UNVERIFIED, UNCERTAIN.
  • Verify-before-write operational gating: write rules combine Scout confidence and Judge verdict to map candidates to WRITE, BLOCK, or REVIEW.
• Evidence grounding
  • Identity-matched web retrieval: web results are identity-checked (not just topically relevant) before using them as evidence to avoid variant/product mismatch.
  • Images are treated as a separate evidence tier (noisy vs textual).
  • Scout abstains (explicit labels like not_found, ambiguous, conflict) rather than hallucinating.
• Evaluation results (selected)
  • Grocery & Alcohol (human-labeled): 500 SKUs, 2,497 SKU–attribute pairs → ScoutAgent extraction accuracy 98.2% at 74.7% attribute coverage.
  • Electronics & Home Improvement: 955 SKUs, 4,990 pairs → ScoutAgent 87.8% attribute coverage; 97.4% of extracted values received PASS or UNVERIFIED from the JudgeAgent.
  • Backbone comparison (ScoutAgent VLM): Gemini 2.5 Flash gave best quality–coverage–cost tradeoff (publication coverage 85.5%, normalized cost 1.0×); other backbones (Gemini 3.5 Flash, GPT-5.4, Claude Sonnet 5) had worse tradeoffs (higher cost or lower judge-supported/publication coverage).
• Deployment & business impact
  • Production: enriched 31 million SKUs across four verticals; impression-weighted enrichment coverage +90.4%.
  • Online A/B (5 weeks, 90% treatment): PDPs surfaced enriched attributes → checkout conversion +0.48% (p=0.034), power users +1.18% (p=0.014), missing/incorrect-item rate −1.08% (p=0.026).
• Limitations & failure modes
  • Electronics/Home Improvement results use JudgeAgent verdicts as scalable signal (not full human validation).
  • Backbone choice affects not only extraction frequency but whether extracted values are sufficiently grounded for publication; errors concentrate in evidence-intensive attributes (unit counts, free-text descriptions).
  • Early binary judge was overly conservative; current four-way taxonomy reduces false rejections but tradeoffs between precision and recall remain.

Data & Methods

• Data sources for enrichment:
  • Seller-provided structured and unstructured fields (titles, descriptions), product images.
  • Syndicated product records from commercial data providers.
  • Agentic, identity-matched web search driven by product identifiers and attribute-specific queries (ReAct-style loop for identity verification).
• Candidate representation (ScoutAgent output includes):
  • attribute a, proposed value v, supporting evidence E (with provenance and source types τ), model-reported confidence q, extraction status z (extracted, not_found, not_applicable, ambiguous, conflict).
• JudgeAgent verdicts and operational policy:
  • Verdicts: PASS (supported), FAIL (contradicted or unsupported), UNVERIFIED (not contradicted but not directly confirmed), UNCERTAIN (conflicting/ambiguous).
  • Publication action d(a) = BLOCK if q < θ or verdict = FAIL; REVIEW if verdict = UNCERTAIN; WRITE if verdict ∈ {PASS, UNVERIFIED}. Threshold θ is tunable.
• Evaluation metrics:
  • Extraction accuracy (human-validated where available), attribute coverage (fraction of requested SKU–attribute pairs with nonempty value), judge-supported rate = fraction of extracted values receiving PASS or UNVERIFIED, publication coverage = fraction of requested attributes receiving publishable verdicts.
• Experimental datasets:
  • Grocery & Alcohol: 500 SKUs, 2,497 pairs, human ground truth used to measure accuracy and analyze Judge behavior.
  • Electronics & Home Improvement: 955 SKUs, 4,990 pairs, JudgeAgent used for scalable adjudication and backbone comparisons.
• LLM/VLM backbones:
  • Both agents implemented with Gemini 2.5 Flash by default; ScoutAgent backbone variants compared (Gemini 3.5 Flash, GPT-5.4, Claude Sonnet 5) with Judge fixed to Gemini 2.5 Flash to isolate operational differences and costs.

Implications for AI Economics

• Operational efficiency and labor substitution
  • TRACE demonstrates that agentic LLM pipelines can replace large-scale manual catalog enrichment with high precision and throughput, reducing ongoing labeling and manual reconciliation costs for marketplaces. The framework still routes ambiguous/conflicting cases to humans, concentrating human effort where it is most valuable.
• Cost–quality tradeoffs in model selection
  • Backbone choice materially affects both coverage and the fraction of values judged publishable; higher-capacity/cost models do not necessarily yield better publication coverage or ROI. Platforms should evaluate model selection using operational metrics (publication coverage, judge-supported rate) not just raw extraction rate.
• Revenue and consumer welfare effects
  • Small relative improvements in conversion (e.g., +0.48%) can translate to economically significant revenue gains at scale. Reduced missing/incorrect items also lower service costs and negative externalities (returns, customer support, reputation).
• Risk management and tail events
  • The verify-before-write JudgeAgent reduces the risk of erroneous or safety-sensitive attribute publication (e.g., allergens). Explicit provenance and the UNVERIFIED/UNCERTAIN taxonomy let a platform tune precision vs recall depending on downstream externalities and regulatory risk.
• Platform-level externalities
  • Better structured attributes improve search, filtering, and personalization systems, increasing match efficiency and potentially altering competitive dynamics among sellers (products that are better-described get more impressions/conversions). This has second-order effects on marketplace allocation and seller incentives to provide high-quality metadata.
• Cost-benefit and investment calculus
  • Economic evaluation should include model inference costs, web retrieval costs, human-review costs for UNCERTAIN cases, and the monetized lift from conversion and reduced errors. TRACE’s production deployment and backbone cost comparisons illustrate that cheaper but well-calibrated VLMs can dominate when judged by publication coverage × cost.
• Policy and governance considerations
  • Platforms must govern provenance, privacy, and potential scraping liabilities when performing identity-matched web retrieval. They should also monitor model drift and maintain the Judge gate and human-in-the-loop auditing to manage downstream economic risks.
• Research & measurement opportunities
  • Quantifying long-run impacts on returns, lifetime customer value, recommendation quality, and seller behavior; optimizing θ (confidence threshold) to maximize economic surplus subject to safety constraints; exploring multi-model juries for judge robustness; formal ROI frameworks to decide when to surface UNVERIFIED vs require PASS.

In short, TRACE is a practical, evidence-grounded agentic pipeline that shows how LLMs can be economically deployed to improve catalog completeness and marketplace outcomes while preserving operational controls to manage risk and cost.