Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch

Dispatch in three-sided marketplaces provides a natural setting for reinforcement learning from world feedback: decisions are evaluated by delayed operational outcomes such as delivery speed, courier utilization, and merchant congestion. We present a deployed reinforcement learning system at DoorDash that adapts dispatch objective weights in a large-scale food-delivery marketplace using delayed signals. Rather than replacing the combinatorial assignment optimizer, a store-level policy learned from logged marketplace data selects a discrete multiplier that shifts the dispatch optimizer's tradeoff between delivery quality and batching efficiency. This interface enables offline policy learning under noisy, delayed, and coupled feedback while preserving production feasibility constraints and operational safeguards. We train a shared value function using centralized offline data and decentralized store-level execution, with Double Q-learning targets and a conservative regularizer to reduce out-of-distribution value overestimation. In a production switchback experiment, the offline-trained policy increases batching and reduces courier-side time costs without degrading customer-facing delivery quality. Results illustrate how world feedback from a live economic and logistics system can be used to safely adapt decision policies online.

Summary

Main Finding

An offline-trained multi-agent RL agent deployed at DoorDash (OWA-RL) successfully adapts store-level objective weights for a production dispatch optimizer using delayed, regional marketplace feedback. By selecting a discrete multiplier on the optimizer’s delivery-speed (ASAP) weight—rather than replacing the optimizer—the system increased batching and reduced courier-side time (CAT and merchant wait CWT) without degrading customer-facing delivery quality in a large-scale, two-week switchback experiment.

Key Points

Problem framing: three-sided marketplace dispatch with delayed, coupled outcomes (customer latency, courier time, merchant congestion). Static global objective weights are brittle to local/time-varying conditions.
Constrained interface: each store-level agent picks one of five discrete multipliers A = {0.8, 0.9, 1.0, 1.1, 1.2} to scale the baseline ASAP objective weight; the combinatorial optimizer still enforces feasibility and assignments.
Reward design: delayed, region-aggregated reward uses ASAP (customer-facing delivery time) and XCAT (excess courier active time beyond direct travel). Reward per attributed decision cycle: rg = −(1/|D|) Σ_j (α·ASAP_j + β·XCAT_j).
Offline MARL setup: decentralized execution (store-level) with centralized offline training using pooled logged transitions and regional rewards to capture network effects.
Learning algorithm: discrete-action Q-function trained with Double DQN targets and a Conservative Q-Learning (CQL) regularizer to reduce overestimation for actions weakly supported in logged data:
- DDQN target y = r + γ Q̄(s′, argmax_a′ Q(s′, a′))
- CQL term: E_{(s,a)} [ log Σ_{a′} exp Q(s, a′) − Q(s, a) ]
Data collection: controlled rollout in two iterations (pure exploration then 50% mixture policy), logging ~3.46M transition samples total from localized 0.5%-region experiments.
Model & training: small MLP Q-network (3-d input → two 16-unit hidden layers → 5 outputs). Training: Adam lr=1e-3, γ=0.99, 30 epochs, batch size 32, target update every 2 epochs; conservative regularizer weight η tuned to improve offline stability.
Production evaluation: global switchback experiment across ~4,000 geographic regions, randomized every 2 hours over two weeks; CUPED variance reduction and cluster-robust p-values used for inference.
Online results (selected):
- All day parts: CAT −1.261 s (p=0.019), CWT −0.856 s (p=0.004), % batched +0.495 pp (p<0.001); ASAP change not significant.
- Dinner: CAT −1.289 s (p=0.042), CWT −1.030 s (p=0.041), % batched +0.600 pp (p=0.010); 20-min lateness improved slightly.
Operational highlights: serving hundreds of millions of daily inferences at ~20s cadence; preserves existing safeguards and optimizer latency/feasibility.

Data & Methods

Formalism: decentralized multi-agent MDP M = (I, S, A, T, R, γ, λ0) where each store i observes s_i^t = (outstanding deliveries d, localized supply-pressure sup, median courier wait time cwt) and selects discrete action a_i^t to set λ_i^t = a_i^t λ0.
Reward attribution timing: join logged dispatch runs with delayed fulfillment outcomes (creation, acceptance, dropoff times) to compute ASAP_j, CAT_j, XCAT_j, then aggregate over region g(i) and decision cycle t.
Offline data pipeline:
- Runs: ~3 runs/min → 4,320 runs/day per region; ~10 stores/run → ~43,200 store-level points/day/region.
- Controlled rollout: 0.5% regions every 2 hours → ≈864k points/day across selected regions; two iterations of ~2 days each → ≈3.46M samples total.
Learning specifics:
- Q-network architecture: Linear(3→16) → ReLU → Linear(16→16) → ReLU → Linear(16→5).
- Objective: L = L_DDQN + η L_CQL. Double DQN to reduce maximization bias; CQL discourages unsupported large Q-values in offline RL.
- Training: initialized from checkpoint, continued 30 epochs; conservative regularizer yields higher Bellman MSE but better offline-to-online stability.
Evaluation & experiment design:
- Switchback design: two-hour intervals, regions randomized to treatment (OWA-RL) or control (baseline weight), approx half regions per arm each interval.
- Metrics: CAT (courier active time), CWT (courier wait time at pickup), % batched, ASAP (customer-facing duration), % 20-min late. CUPED variance reduction; p-values clustered at region-hour buckets.
Safety and monitoring: guardrails for customer experience, cancellations, carryover; additional monitoring for state/action/reward distributions and distribution shift.

Implications for AI Economics

Practical path to RL in marketplaces: Shows a low-risk, high-leverage pattern—use RL to adapt objective/priority weights while retaining a tested combinatorial optimizer. This reduces operational risk and deployment friction compared to end-to-end RL that directly controls assignments.
Efficiency gains with limited behavioral change: Small per-trip reductions in courier-side time (seconds) and modest increases in batching can aggregate to material operational cost savings at scale (lower courier time costs, higher throughput). These are economically meaningful when multiplied across millions of deliveries.
Redistribution and welfare effects: Adaptive weighting shifts allocation of waiting/travel burden across stakeholders. Even when customer latency is preserved on average, increased batching may change variance in wait times, affect merchant congestion, and alter courier scheduling. Platform-level welfare gains must be assessed alongside distributional effects across couriers, merchants, and customers.
Dynamic equilibrium & externalities: The system optimizes under current marketplace dynamics; sustained policy changes can induce supply-side responses (courier availability, acceptance behavior), demand-side changes, or routing adjustments by partners. These second-order equilibrium effects require ongoing monitoring and possibly integrated models (e.g., supply response, pricing).
Measurement and identification challenges: Delayed, regional aggregation improves network-effect attribution but complicates credit assignment to store-level actions and increases noise. Robust causal evaluation (switchbacks, CUPED) and conservative offline RL methods are essential to avoid spurious gains from distribution shift or unsupported actions.
Governance, labor and regulation: Automating objective-weight adaptation affects courier workloads and merchant congestion—requiring guardrails, transparency, and potentially policy/regulatory scrutiny. Platforms should pair such deployment with monitoring, rollback criteria, and impact assessments on labor outcomes.
Generalizability: The approach (low-dimensional control interface + offline conservative RL + randomized real-world evaluation) is applicable to many two-/three-sided marketplaces and logistics systems that have existing optimizers and delayed operational feedback.
Research directions for AI economics:
- Integrate behavioral supply models to predict long-run supply response and optimize for equilibrium welfare rather than one-shot operational metrics.
- Jointly learn objective-weight policies with pricing, incentives, or routing layers to internalize broader economic feedback.
- Develop causal RL diagnostics and methods to better handle noisy, delayed, networked rewards and strategic agents.
- Study distribution-shift detection and automated retraining thresholds to maintain robustness.

Caveats - Reported gains are modest per-trip but scale-dependent; long-run effects may differ if marketplace dynamics shift. - Offline training reliability depends on stability of post-deployment dynamics; CQL mitigates but does not eliminate OOD risks. - Regional reward aggregation masks some local effects and can hide heterogeneity; careful local analysis and guardrails remain necessary.

Bottom line: The paper demonstrates a practical, scalable pattern for deploying RL in large marketplaces by restricting RL control to an interpretable objective-weight interface, using conservative offline training on delayed, regional rewards, and validating effects with randomized switchback experiments—yielding measurable efficiency gains while preserving customer quality.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The paper reports results from a live production switchback experiment (stronger than pure offline evaluation), demonstrating improvements in batching and courier-side time costs without degrading delivery quality; however, the writeup lacks key details on randomization/balance, statistical uncertainty, experiment duration, spillovers/carryover effects, and heterogeneity, which prevents grading the causal evidence as high. Methods Rigormedium — Methodologically the work uses appropriate and state-of-the-art RL safeguards (Double Q-learning, conservative regularizer), preserves the production combinatorial optimizer via a constrained interface, and trains on large centralized logs; nevertheless, the paper provides limited detail on hyperparameter selection, off-policy evaluation diagnostics, handling of delayed/coupled feedback, robustness checks, and the exact experimental assignment protocol, leaving important implementation and inference risks unaddressed. SampleLarge-scale DoorDash marketplace logged data at the store/order level used to train a shared value function; execution at run-time is decentralized to stores which select discrete multipliers for the dispatch optimizer; evaluation is based on a subset of stores/regions exposed in a production switchback experiment, with outcome metrics including batching rates, courier-side time costs, delivery speed/quality, courier utilization and merchant congestion. Themesproductivity human_ai_collab org_design IdentificationCausal effects assessed via a production switchback experiment that alternately exposes stores/regions to the offline-trained store-level RL multiplier policy versus the baseline dispatcher; offline policy learning uses logged marketplace data with Double Q-learning targets and a conservative regularizer to limit out-of-distribution value overestimation. GeneralizabilitySingle-platform (DoorDash) and food-delivery context — results may not generalize to other types of marketplaces or industries., Relies on a specific combinatorial dispatch optimizer and a store-level multiplier interface; different optimizer architectures may not admit the same intervention., Operational context (market density, courier workforce composition, local regulations) may alter effects in other cities/countries., Switchback experimental design may be sensitive to seasonal/time confounders and carryover/spillover between treated and control stores., Offline training and value-regularization choices are implementation-dependent; transfer to other data regimes or less logged environments is uncertain.

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
We present a deployed reinforcement learning system at DoorDash that adapts dispatch objective weights in a large-scale food-delivery marketplace using delayed signals. Adoption Rate	positive	deployment and adaptation of RL system to marketplace delayed signals	Reading fidelity high Study strength medium	not reported 0.48
A store-level policy learned from logged marketplace data selects a discrete multiplier that shifts the dispatch optimizer's tradeoff between delivery quality and batching efficiency. Task Allocation	mixed	tradeoff between delivery quality and batching efficiency (via discrete multiplier selection)	Reading fidelity high Study strength low	not reported 0.24
This interface enables offline policy learning under noisy, delayed, and coupled feedback while preserving production feasibility constraints and operational safeguards. Other	positive	ability to perform offline policy learning while preserving feasibility/safeguards	Reading fidelity high Study strength low	not reported 0.24
We train a shared value function using centralized offline data and decentralized store-level execution. Other	positive	shared value function trained from centralized offline data	Reading fidelity high Study strength low	not reported 0.24
We use Double Q-learning targets and a conservative regularizer to reduce out-of-distribution value overestimation. Other	positive	out-of-distribution value overestimation (reduction)	Reading fidelity high Study strength medium	not reported 0.48
In a production switchback experiment, the offline-trained policy increases batching. Task Allocation	positive	batching rate/level	Reading fidelity high Study strength medium	not reported 0.48
In a production switchback experiment, the offline-trained policy reduces courier-side time costs. Task Completion Time	positive	courier-side time costs	Reading fidelity high Study strength medium	not reported 0.48
In a production switchback experiment, the offline-trained policy reduces courier-side time costs without degrading customer-facing delivery quality. Output Quality	null_result	customer-facing delivery quality	Reading fidelity high Study strength medium	not reported 0.48
Results illustrate how world feedback from a live economic and logistics system can be used to safely adapt decision policies online. Organizational Efficiency	positive	use of world feedback to adapt policies safely	Reading fidelity high Study strength medium	not reported 0.48

A reinforcement-learning policy at DoorDash nudges the dispatch optimizer to increase batching and cut courier time costs without worsening delivery quality; the approach adapts objective weights via a constrained store-level multiplier learned from logged marketplace feedback.