1 cumulative citations
View corpus contextA centralized policy gradient approach (MAPPO) consistently earns the highest and most stable profits in a simulated competitive retail pricing environment, while MADDPG spreads profits more evenly across sellers; independent learners trail on both returns and stability.
Multi-Agent Reinforcement Learning for Dynamic Pricing: Balancing Profitability,Stability and Fairness
In a retail marketplace simulator calibrated to real data, MAPPO yields the highest average profits with low variance, MADDPG produces the fairest profit distribution, and independent DDPG performs worse across profit and stability metrics.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Dynamic pricing in competitive retail markets requires strategies that adapt to fluctuating demand and competitor behavior. In this work, we present a systematic empirical evaluation of multi-agent reinforcement learning (MARL) approaches-specifically MAPPO and MADDPG-for dynamic price optimization under competition. Using a simulated marketplace environment derived from real-world retail data, we benchmark these algorithms against an Independent DDPG (IDDPG) baseline, a widely used independent learner in MARL literature. We evaluate profit performance, stability across random seeds, fairness, and training efficiency. Our results show that MAPPO consistently achieves the highest average returns with low variance, offering a stable and reproducible approach for competitive price optimization, while MADDPG achieves slightly lower profit but the fairest profit distribution among agents. These findings demonstrate that MARL methods-particularly MAPPO-provide a scalable and stable alternative to independent learning approaches for dynamic retail pricing.
Summary
Main Finding
MAPPO (on-policy CTDE PPO) delivers the best trade-off for competitive retail dynamic pricing in this simulated, data-driven marketplace: it achieves the highest mean profits with low variance and good fairness. MADDPG yields slightly lower profits but the fairest profit distribution. MASAC is unstable in this environment and collapsed to negative profits, while independent DDPG (IDDPG) is less stable and produces lower average profit than CTDE methods.
Key Points
- Algorithms compared: MAPPO, MASAC, MADDPG (CTDE variants) and an Independent DDPG (IDDPG) baseline.
- Benchmark criteria: average profit, training stability (std over seeds), sample efficiency, fairness (Jain’s index, Gini), and competitive behaviors (price volatility, undercutting).
- Results summary (from Table II in the paper):
- MAPPO: Average profit = 0.91, std = 0.72, Jain’s index = 0.94, Gini = 0.12.
- MADDPG: Average profit = 0.79, std = 0.69, Jain’s index = 0.96, Gini = 0.11.
- IDDPG: Average profit = 0.62, std = 1.12, Jain’s index = 0.78, Gini = 0.29.
- MASAC: Average profit ≈ −0.007, std = 0.02, poor fairness metrics; collapsed early in training.
- Interpretations:
- MAPPO: highest mean profit, low variance across seeds → reproducible and stable.
- MADDPG: most equitable profit distribution → preferred when fairness across sellers matters.
- MASAC: high exploratory potential, but sensitive to hyperparameters and susceptible to Q-estimation / entropy tuning problems in this competitive pricing setting.
- IDDPG: simple but non-stationarity among independent learners causes high variability and suboptimal outcomes.
Data & Methods
- Data
- Source: trimmed UCI Online Retail dataset (~540k rows originally); filtered to ~19k rows covering top 50 SKUs by sales.
- Demand model per SKU: CatBoost regressors (monthly aggregation). Validation: R2 ≈ 0.6547, RMSE ≈ 723.01, MAPE ≈ 1.24.
- Chronological split: 160 training periods / 40 validation periods.
- Environment / Simulation
- MarketplaceEnv parameterized from fitted demand models.
- N = 3 sellers (agents); each episode T = 24 steps (monthly pricing decisions, 2 years).
- Actions: continuous price adjustments (normalized to [−1,1]) mapped to ±30% of reference price.
- Demand allocation: softmax market-share model with competition intensity β = 10.
- Unit cost: modeled as 70% of selling price; profits computed accordingly.
- Stochasticity: Gaussian noise added to demand (σ ≈ 730, clipped at 3σ) based on model residuals.
- Algorithms & Architectures
- MAPPO: on-policy CTDE PPO with centralized value network, GAE (λ=0.95), clip ϵ=0.2.
- MASAC: off-policy entropy-regularized actor-critic (two Qs, target networks).
- MADDPG: CTDE deterministic actor-critic (centralized Q per agent).
- IDDPG: independent DDPG per agent (baseline).
- Neural nets: actors = 2×128 FC layers (Tanh); critics (when joint) = 256–256 FC (ReLU).
- Training & Evaluation
- 400 training episodes per seed, 10 random seeds.
- Off-policy methods use replay buffer size 1e5 transitions.
- Shared hyperparameters: lr = 3e-4, γ = 0.99, minibatch = 128. PPO epochs = 4.
- Evaluation: every K=20 episodes, deterministic evaluation runs (3 episodes) and metrics averaged. Metrics tracked include average profit, std across seeds, sample efficiency (episodes to reach 80% of IDDPG asymptotic profit), Jain’s index, Gini, price volatility, undercutting frequency, market-share churn.
- Hardware: CPU-only (Intel i7, 32 GB RAM), PyTorch 2.2.1, Python 3.10.
Implications for AI Economics
- Algorithmic recommendation for practitioners:
- Use MAPPO (CTDE PPO) when the primary objective is profit maximization with reproducible training and low variance across deployments. Its stability makes it a practical choice for production dynamic pricing systems.
- Use MADDPG when equitable outcomes between competing sellers are important (e.g., marketplace fairness constraints or regulatorily-motivated parity).
- Be cautious with MASAC in multi-seller pricing without careful tuning; entropy-regularized, off-policy methods can collapse due to Q-estimation bias and sensitive coefficient settings in adversarial/multi-agent settings.
- Avoid naive independent learners (IDDPG) in competitive multi-seller contexts: non-stationarity among independently trained agents produces higher variance and lower profits.
- Market and policy considerations:
- Automated dynamic pricing via MARL can materially change competitive dynamics—higher profitability and lower price volatility under certain algorithms—but introduces risks of emergent collusion-like outcomes. Regulators and platform designers should evaluate welfare impacts (consumer surplus), collusion risk, and transparency requirements.
- Fairness trade-offs: maximizing aggregate profit (MAPPO) can slightly reduce equity compared to MADDPG; platform rules or incentives might be needed where balanced seller outcomes are desired.
- Research and deployment directions:
- Scale and realism: replicate experiments with more sellers (N ≫ 3), richer product assortments, heterogeneous inventories, and richer customer choice models to validate transferability.
- Field evaluation: run controlled A/B tests in production to measure GMV, conversion, and consumer effects; simulation success does not guarantee field performance.
- Robustness and safety: study hyperparameter sensitivity (especially for MASAC), adversarial behaviors, and equilibrium properties; incorporate constraints (e.g., no-excessive-variability, minimum margin) to align learned policies with business/regulatory constraints.
- Economics analysis: combine MARL benchmarks with theoretical work on equilibria in continuous-action pricing games to better understand stability and welfare properties.
- Limitations to note:
- Small agent count (N=3) and trimmed dataset may limit generality.
- Simulation driven by fitted demand models—model misspecification could change results in real markets.
- CPU-only experiments and limited hyperparameter sweeps mean MASAC or other algorithms might perform better under different compute/resources or tuning.
Overall, this paper provides empirical evidence that CTDE on-policy methods (MAPPO) are a practical, stable approach for multi-agent dynamic pricing problems in retail, while highlighting fairness vs. profitability trade-offs and the need for careful algorithm selection and validation before deployment.
Assessment
Paper Typedescriptive
Evidence Strengthlow — Findings are based on simulations calibrated to historical retail data rather than field experiments or observational causal inference; results establish algorithmic performance in a lab environment but do not demonstrate real-world causal impacts on firms, markets, or consumers.
Methods Rigormedium — The study benchmarks multiple standard MARL algorithms (MAPPO, MADDPG, IDDPG), evaluates across random seeds, and reports profit, variance, fairness, and training efficiency, which indicates careful empirical practice; however, it lacks real-world deployment, limited sensitivity analyses and robustness checks across diverse market specifications, and details on calibration are not provided here.
SampleA simulated multi-agent retail marketplace calibrated from real-world retail data; agents represent competing retailers using MARL (MAPPO, MADDPG, IDDPG) to set dynamic prices; evaluation uses multiple random seeds and reports agent-level profits, profit variance (stability), fairness of profit distribution, and training efficiency metrics.
Themesinnovation adoption
GeneralizabilitySimulation-based results may not generalize to live retail markets with richer strategic behavior and noise, Calibration may reflect a specific product category, geography, or time period limiting applicability to other markets, Agent heterogeneity in the simulation may be limited compared with real firms (size, inventory, multi-product pricing), Consumer demand model and competitor response functions in the simulator may omit real-world features (search frictions, brand effects, stockouts), Regulatory, logistical, and human managerial constraints present in real deployments are not captured, Scalability and latency constraints in large-scale marketplaces may alter algorithm performance
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| MAPPO consistently achieves the highest average returns with low variance, offering a stable and reproducible approach for competitive price optimization. Firm Revenue | positive | average returns (profit) and variance |
Reading fidelity
high
Study strength
medium
|
not reported
|
| MADDPG achieves slightly lower profit but the fairest profit distribution among agents. Inequality | mixed | profit (average returns) and fairness of profit distribution |
Reading fidelity
high
Study strength
medium
|
not reported
|
| MARL methods—particularly MAPPO—provide a scalable and stable alternative to independent learning approaches for dynamic retail pricing. Organizational Efficiency | positive | dynamic retail pricing performance (profitability and training stability/scalability) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We use a simulated marketplace environment derived from real-world retail data. Other | null_result | simulation data source / environment construction |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We benchmark these algorithms against an Independent DDPG (IDDPG) baseline, a widely used independent learner in MARL literature. Other | null_result | algorithmic benchmarking against IDDPG |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We evaluate profit performance, stability across random seeds, fairness, and training efficiency. Other | null_result | profit performance; stability (across random seeds); fairness (profit distribution); training efficiency |
Reading fidelity
high
Study strength
medium
|
not reported
|