Integrating LLM-generated user profiles into DiDi's dispatcher raised prediction AUC by up to 6.14% and produced a measurable lift in platform revenue and service quality (≈+0.47% GMV, +0.33% completion rate) in a 14‑day production A/B test. The deployment also reduced cancel-before-accept rates, showing that utility-aligned LLM profiling can deliver real economic value when engineered for low-latency production use.

Main Finding

ProfiLLM is an agentic, production-deployed pipeline that uses offline LLM-based user profiling to improve ride-hailing dispatch outcome prediction and platform utility. By mining platform-scale global knowledge, clustering users to address long-tail sparsity, and optimizing profile generation for downstream prediction utility (via a rule-based utility proxy and DPO fine-tuning), ProfiLLM achieved up to +6.14% relative AUC in outcome prediction, up to +4.35% GMV improvement in dispatch simulation, and a 14-day A/B lift of +0.47% GMV, +0.33% completion rate, and −0.82% cancel-before-accept rate — all while adding sub-millisecond online overhead and zero online LLM inference.

Key Points

• Problem targeted: improve per order-driver (OD) outcome prediction (acceptance, cancellations, completion) used by real‑time matching, where structured features miss contextual behavioral signals.
• Three deployment constraints addressed together:
  • Historical logs vastly exceed LLM context windows and compute.
  • Severe long-tail (majority of passengers have very few orders).
  • LLM-generated fluent profiles do not necessarily improve downstream prediction utility.
• System design (strict offline–online contract):
  • All LLM reasoning and heavy compute run offline.
  • Online dispatcher only performs deterministic cluster assignment and a lookup of precomputed cluster-level profile embeddings; no online LLM calls; online overhead <0.01 ms per OD pair.
• Two offline modules:
• Tool-Augmented Global Knowledge Mining: an LLM agent with 27 composable analytical tools (organized in six categories) runs an Explore → Deepen → Validate → Synthesize workflow to produce: - Global behavioral knowledge K (temporal patterns, weather impacts, causal findings). - Interpretable user-clustering rules A (boolean rules mapping histories to clusters). - Regional supply–demand priors R (grid & time-slot statistics).
• Utility-Aligned Profile Exploration: - Partition users by rules A to form clusters (aggregating data to overcome long-tail). - For each cluster generate K candidate structured profiles (ANALYSIS, PROFILE, LOGIC). - Use LOGIC rules as a lightweight utility proxy: apply rules to historical OD pairs, blend rule outputs with base model outputs (convex blend with coefficient λ), compute fused AUC improvement Δ to rank profiles. - Iteratively refine top candidates; create preference pairs to DPO-finetune the LLM profile generator so generation is aligned to downstream utility. - Emit one profile per cluster, encoded into a frozen text-encoder embedding for online use.
• Practical deployment: implemented at DiDi on a production dispatcher (2-second cycles, ~200 ms total latency budget) with measurable economic impact.

Data & Methods

• Data scale & distribution:
  • Example: one city produced ~44.3M dispatching records over 38 days.
  • Severe long-tail: ~96% of passengers had ≤30 orders in that window; drivers had higher frequency but passenger sparsity dominates.
• Prediction tasks:
  • A multi-task production predictor outputs calibrated probabilities for four funnel events: driver acceptance, post-acceptance driver cancellation, post-acceptance passenger cancellation, and completion.
• Tools and analytics:
  • 27 analytical tools implement statistical summaries, spatiotemporal aggregation, clustering, hypothesis testing, causal probes, weather/temporal impact analysis, regional supply–demand computations, etc.
  • Agent composes tools programmatically guided by the Explore–Deepen–Validate–Synthesize paradigm.
• Profile generation & evaluation:
  • Cluster-level aggregated histories H_a form prompts together with global K to generate K diverse candidate profiles per cluster.
  • Each profile contains an executable LOGIC rule set; rules are applied to historical OD pairs to obtain rule-based predictions.
  • Blended prediction: y_fused = (1−λ)y_base + λy_logic; compute fused AUC to measure utility.
  • Best profiles are iteratively refined and used to create preference pairs to DPO fine-tune the LLM generator.
• Online serving:
  • Cluster-assignment rules are boolean functions φ_a(H_u) evaluated online to map user to cluster a.
  • Precomputed cluster embeddings e_a are fetched and concatenated with structured features for the production predictor; matching proceeds unchanged.
• Reported outcomes:
  • Up to +6.14% relative AUC improvement on outcome prediction.
  • Up to +4.35% GMV gain in dispatch simulator.
  • 14‑day A/B in City A: +0.47% GMV, +0.33% Completion Rate, −0.82% Cancel-Before-Accept.

Implications for AI Economics

• Direct revenue and efficiency uplift: measurable increases in GMV and completion rate, and reduced cancellations, imply improved platform matching efficiency and short-run revenue. Even small percentage improvements at platform scale can be economically significant.
• Cost-effective LLM integration: by confining LLMs to offline batch pipelines and shipping only compact embeddings and interpretable rule-sets to the online system, ProfiLLM demonstrates a practical pattern for extracting high-value semantic signals from LLMs without incurring prohibitive latency or per-inference costs. This separation improves marginal cost economics of applying frontier LLMs to high-throughput systems.
• Long-tail mitigation via clustering: aggregating low-frequency users into behaviorally meaningful clusters is an economically efficient way to obtain personalization-like gains without per-user profiling costs; cluster granularity trades off between utility gains and storage/maintenance costs.
• Utility alignment matters: optimizing profile generation for downstream predictive utility (not fluency) is essential — the DPO-driven alignment approach is an example of economically motivated model tuning that targets the platform objective directly.
• Operational & policy considerations:
  • Re-training cadence and offline compute costs must be balanced against the revenue uplift; a quantitative cost-benefit analysis (offline LLM compute + tooling vs incremental GMV) is necessary for investment decisions.
  • Interpretability (rule-based clusters and LOGIC) helps auditing, regulatory compliance, and operator trust — important in regulated transport markets.
  • Behavioral feedback loops and externalities: improved matching could change driver/passenger behavior (e.g., drivers gaming patterns, strategic repositioning), so the platform should monitor dynamics and consider causal A/B tests to separate prediction gains from induced behavioral change.
  • Fairness and market design: cluster-based policies might inadvertently privilege or disadvantage certain user groups or geographies; economics teams should evaluate distributional impacts (inequality of service, geographic coverage).
• Directions for further economic research:
  • Causal evaluation of profiling interventions (e.g., randomized experiments tying profile-informed matching to pricing and incentive design).
  • Dynamic cluster adaptation and pricing interactions — how do cluster-aware matches interact with surge pricing and labor supply?
  • Cost-benefit modeling of offline LLM computation frequency, model size, and DPO fine-tune iterations relative to measurable economic KPIs (GMV, retention, driver income).
  • Generalization to other two-sided platforms (food delivery, logistics) where semantic behavioral signals could similarly improve matching efficiency.

Summary: ProfiLLM provides a scalable, utility-aligned blueprint for bringing LLM semantic profiling into latency‑sensitive marketplaces. Its offline-agent + rule/embedding interface reconciles LLM capabilities with production constraints and demonstrates economically meaningful gains, while raising further operational and economic questions about costs, dynamics, and fairness.