0 cumulative citations
View corpus contextDeep-Q pricing agents in a congestible-market simulation can learn tacit collusion: longer memory and slower exploration lead to higher-than-competitive prices, although competitive pricing re-emerges under larger numbers of firms in the model.
Multi-Agent Reinforcement Learning in Markets with Congestion
In a simulated Bertrand congestion market, independent deep Q-learning agents can converge to supra-competitive (collusive) prices depending on memory (state K), exploration decay, and number of firms, with competition restored as N increases in the tested settings.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
1 cumulative citations
View corpus contextThis paper investigates multi-agent reinforcement learning (MARL) in settings where firms compete for customers using congestible resources. We consider Bertrand competition in which firms compete by announcing prices and customers choose among firms based on both price and congestion. The relationship between price, congestion and the quantity of customers willing to accept service is governed by an unknown inverse demand curve, which firms must learn through experience. Each firm is modeled as a self-interested learning agent that chooses its price to maximize profit. A growing literature has shown that independently learning MARL agents can develop tacitly collusive behavior. We examine how such behavior emerges in markets with congestible resources. Our results provide insight into how learning dynamics, state representation, and strategic interaction jointly shape competition, with implications for both economic learning and the design of learning-enabled markets.
Summary
Main Finding
Independent deep Q-learning agents competing in a Bertrand pricing game with a shared, congestible resource can learn supra-competitive pricing. The emergence and extent of tacit collusion depend systematically on (i) the agents’ state representation (history length), (ii) their exploration decay schedule, and (iii) the number of firms. Congestion externalities change strategic incentives relative to standard Bertrand models and shape what learning dynamics produce.
Key Points
- Model setup
- N firms announce per-user service prices pi. Users experience a delivered price = pi + congestion cost ˆg(x), where ˆg depends on total load X = Σ xi (Wardrop allocation determines xi).
- Market demand is given by an unknown inverse demand P(X) (agents do not observe P), and customers choose lowest delivered price; ties split demand evenly.
- Firms maximize revenue ri = pi xi (costs normalized to zero).
- Learning formulation
- Each firm is an independent deep Q-network (DQN) agent (separate value nets and replay buffers). State = joint price actions from the previous K rounds (finite history); action space = discrete set of prices; reward = realized revenue.
- Agents treat others as part of the environment (no centralized training or communication).
- Empirical benchmarks and choices
- For analysis/benchmarks the authors use linear inverse demand P(X) = 1 − X and normalize capacity We = 1.
- Discrete price grid: {0.05, 0.15, …, 0.95}.
- Derived full-information Nash and collusive (joint revenue-maximizing) benchmarks for comparison.
- Main empirical observations
- N = 2, K = 2, high exploration decay (δ = 0.99): agents often converge to a Pareto-dominant Nash equilibrium (p = 0.15), not the fully competitive minimum price (0.05).
- Increasing history length K tends to increase the incidence of higher-than-Nash prices (stronger tacit collusion), though often not reaching the full collusive benchmark.
- Lowering exploration decay (faster reduction of exploration; e.g., δ = 0.9) can produce significantly higher learned prices (example: learned prices ≈ 0.35 for N = 2), indicative of tacit collusion driven by repeated-play/punishment dynamics.
- Increasing the number of firms can eliminate higher-price Nash equilibria: for their parameters, N = 3 leaves only the fully competitive equilibrium (p = 0.05), and agents converge to it (though with different exploration schedules higher prices can sometimes re-emerge).
- Conceptual distinctions
- The authors distinguish coordination on a Pareto-dominant static Nash equilibrium (mild coordination) from repeated-play tacit collusion that sustains prices above any static Nash through intertemporal reward/punishment dynamics. Larger K and lower exploration encourage the latter.
Data & Methods
- Analytical model
- Congestion modeled via a convex increasing function ˆg(X/We). Customer allocation follows Wardrop equilibrium conditions: served firms have equal delivered prices equal to P(X).
- Full-information Nash and collusive optima derived under linear demand P(X) = 1 − X for benchmarking.
- Simulation/learning experiments
- Agents: independent DQNs with experience replay and target networks.
- State: last K joint price actions (K varied including K = 0, bandit case).
- Actions: discrete price grid (0.05 to 0.95 in increments of 0.1).
- Reward: realized revenue xi · pi; tie-breaking: equal split of demand when delivered prices equal.
- Parameters studied: number of firms N (2 and 3 shown), history length K, exploration decay rate δ. Capacity We = 1 in experiments.
- Outcomes: episode-level learned prices and convergence behavior compared to Nash and collusive benchmarks. Figures illustrate time series of prices and aggregated outcomes across parameter sweeps (K, δ, N).
- Limitations noted by authors
- Experiments use a specific linear demand (for benchmarks), discrete price grid, small N, and normalized zero marginal cost. Agents are independent DQNs — other algorithms or richer state/action spaces may alter results.
Implications for AI Economics
- Design and regulation
- Algorithm design choices (state representation, memory of past actions, exploration schedules) materially affect the risk of tacit collusion. Regulators and market designers should consider these as potential levers or risk factors when assessing algorithmic pricing.
- Limiting the observability of competitors’ past actions or restricting history-based state information could reduce the channels that sustain repeated-play collusion.
- Enforcement and monitoring should account for collusion mechanisms that do not rely on explicit communication but instead on learned strategies exploiting repeated interactions and congestion externalities.
- Market structure and congestion
- Shared-resource (congestion) externalities change strategic incentives relative to homogeneous-product Bertrand models; equilibrium multiplicity and the presence/absence of collusive outcomes depend on these externalities and the number of firms.
- In markets with congestion, learning algorithms might converge to equilibria that are neither the fully competitive benchmark nor the full cooperative benchmark; predictions require considering both externalities and learning dynamics.
- Research directions
- Tests for algorithmic collusion should incorporate dynamic features (history dependence, exploration schedules) and externalities (congestion) rather than relying solely on static pricing comparisons.
- Further work needed on larger markets, alternative demand forms, continuous action spaces, other learning algorithms (policy gradients, centralized training), and mechanism designs that mitigate implicit collusion while preserving efficiency.
- Broader contribution
- Extends the algorithmic-collusion literature into congestion-based markets, showing how multi-agent learning interacts with externalities to produce varied competitive outcomes.
Assessment
Paper Typedescriptive
Evidence Strengthlow — Results are based on simulation experiments with synthetic demand (P(X)=1−X), a discrete price grid, and independent DQN agents; there is no real-world data, limited parameter sweeps, and no rigorous statistical validation or external robustness checks to establish general causal claims.
Methods Rigormedium — The paper uses standard MARL methods (independent DQN, replay buffers, target networks) and provides analytic benchmarks for comparison, but experimental reporting is thin (few details on network architectures, number of random seeds, training stability, statistical variability, and hyperparameter sensitivity), and the empirical scope is narrow.
SampleSimulation experiments with N symmetric firms competing for non-atomic customers over a shared congestible resource (capacity We normalized to 1). Experiments use an inverse demand P(X)=1−X, discrete price set {0.05,0.15,...,0.95}, and independent Deep Q-Network agents whose state is the joint price history of length K; key parameters varied include K, exploration decay rate δ, and number of firms (N). Outcomes are learning trajectories and final prices/revenues relative to analytic Nash and collusive benchmarks. No real-world or observational data.
Themesgovernance org_design
GeneralizabilityRelies on a specific linear inverse demand (P(X)=1−X) which may not capture real markets, Discrete and small action space (coarse price grid) limits applicability to continuous pricing contexts, Simulations focus on small N (examples N=2,3); results may not generalize to larger markets, Symmetric firms, normalized capacity, and non-atomic customer assumption simplify real-world heterogeneity, Dependent on specific DQN implementations and unexplored hyperparameter choices (architecture, seeds, training time), No empirical validation with field or observational market data
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| For two firms using a history of the previous two rounds (K=2) and an exploration decay rate of δ=0.99, the learning agents eventually select the Pareto-dominant Nash equilibrium with price p=0.15. Market Structure | positive | Learned equilibrium price and whether agents coordinate on a Pareto-dominant Nash equilibrium |
Reading fidelity
high
Study strength
low
|
n=2
p=0.15
|
| In the stateless bandit setting (K=0), the two agents consistently select a Pareto-dominant Nash equilibrium rather than the fully competitive price. Market Structure | positive | Equilibrium selection and learned pricing behavior |
Reading fidelity
high
Study strength
low
|
n=2
|
| As the state-history length K increases, the agents sometimes coordinate on higher prices, and the amount of collusion appears to increase, although it remains below the collusive benchmark. Market Structure | positive | Degree of tacit collusion, measured through learned prices relative to the collusive benchmark |
Reading fidelity
high
Study strength
low
|
n=2
|
| With two firms, K=2, and a lower exploration decay rate of δ=0.9, the learned prices for both agents are around 0.35, exceeding the Nash-equilibrium benchmark and suggesting tacit collusion. Market Structure | positive | Learned prices relative to the Nash-equilibrium benchmark |
Reading fidelity
high
Study strength
low
|
n=2
prices around 0.35
|
| For three firms with K=1 and δ=0.99, the only Nash equilibrium is the fully competitive price p=0.05, and the learning agents converge to that price. Market Structure | null_result | Converged market price and equilibrium selection |
Reading fidelity
high
Study strength
low
|
n=3
p=0.05
|
| The Pareto-dominant Nash equilibrium observed with two firms does not survive when the number of firms increases to three, leaving only the fully competitive price equilibrium. Market Structure | negative | Persistence of firm-favorable equilibrium pricing as the number of firms increases |
Reading fidelity
high
Study strength
low
|
n=3
|
| The state representation, exploration decay rate, and number of firms all affect the emergence of collusive prices in congestion-based markets. Market Structure | mixed | Emergence and intensity of collusive pricing |
Reading fidelity
high
Study strength
low
|
n=3
|