Decentralized, unaware learning algorithms can reach the full-information Nash equilibrium in linear‑quadratic dynamic games, but limited information slows learning and cuts profits and welfare in sticky-price Cournot markets; publishing aggregate output materially speeds convergence and reduces those losses.

Main Finding

Radically uncoupled agents—who observe only a common state and their own actions and thus treat opponents as unobserved noise—can nevertheless learn to play the complete-information feedback Nash equilibrium in infinite-horizon, nonzero-sum linear–quadratic (LQ) stochastic games. The paper introduces a decentralized multi‑agent ϵ‑greedy iterated least‑squares (ILS) algorithm and proves almost‑sure convergence (and finite‑sample high‑probability bounds). The convergence rate depends on the intrinsic stability of the Nash equilibrium and the agents’ learning hyperparameters; under sufficiently strong stability and optimal single‑agent exploration decay, the learned linear policies attain the single‑agent benchmark last‑iterate rate (˜O(t−1/4)). Applied to dynamic Cournot competition with sticky prices, limited-information learning slows convergence, reduces firm profits, and (under high price stickiness) lowers total surplus and raises market concentration; public disclosure of aggregate output substantially accelerates learning and mitigates welfare losses.

Key Points

• Problem setup

  • M‑player infinite‑horizon LQ stochastic game with common linear state xt affected by all players’ controls.
  • Each player only observes the common state history and her own past actions (radically uncoupled information). Players do not observe other players’ actions or the true system matrices.
  • Players adopt linear feedback strategies um_t = −Fm xt + fm + αm v^m_t with bounded Fm, fm and decaying exploration scale αm.

• Learning algorithm

  • Multi‑agent ϵ‑greedy iterated least‑squares (ILS): each player (independently and asynchronously) runs epochwise least‑squares to estimate a misspecified single‑agent model (omitting other players’ controls), computes the optimal linear policy for that perceived model, and injects decaying exploration noise within epochs. Epoch lengths grow over time.
  • No coordination, heterogeneous update schedules, and only private/state observations required.

• Theoretical contributions

  • Identification challenge: players cannot learn true system parameters (they misattribute opponents’ impact to noise). The authors show the learned parameters nevertheless capture aggregate strategic effects and effectively produce noisy best responses to others’ current strategies.
  • Convergence guarantees: under a stability condition on the complete‑information Nash equilibrium, the algorithm converges almost surely to that Nash equilibrium. Finite‑sample, high‑probability bounds are also provided.
  • Convergence rate characterization: rate depends on a stability constant ζ∈(0,1) (intrinsic stability of Nash) and the epoch growth factor λ and exploration schedule. When the equilibrium is sufficiently stable and players use the single‑agent optimal exploration decay, the last‑iterate rate matches the single‑agent ˜O(t−1/4). Otherwise the rate is slower and determined by the game stability (the paper quantifies this dependence).

• Applications & empirical findings

  • Dynamic Cournot with sticky prices: learning under radical uncoupling validated in duopoly simulations.
  • Welfare and market structure: during learning, firms’ profits decline vs. complete‑information equilibrium in both low and high price‑stickiness regimes; total surplus declines persistently only under high stickiness, and market concentration increases (more so under high stickiness).
  • Policy lever: publicly revealing aggregate market output speeds convergence and reduces welfare losses.
  • Robustness: algorithm remains effective in a mildly nonlinear inverse demand example, converging to a good linear approximation.

• Additional properties and caveats

  • Algorithm tolerates asynchronous and random update schedules.
  • If the Nash equilibrium is not globally stable, learning dynamics can exhibit persistent oscillations.
  • Assumptions include sub‑Gaussian noise, bounded admissible strategy sets, existence/feasibility conditions for perceived single‑agent problems, and a technical stability condition for convergence.

Data & Methods

• Formal model

  • General M‑player LQ stochastic game: xt+1 = A0 + A xt + Σm Bm um_t + ωt with quadratic (possibly cross‑term) costs Ym(xt, xt+1, um_t).
  • Each player m uses a perceived (misspecified) single‑agent linear model xt+1 = Θ_m^T z^m_t + ω'_t where z^m_t = [1, xt, um_t], omitting other players’ controls.

• Algorithmic method

  • Iterated least‑squares (ILS) per player: fit perceived model parameters from own state & control history over expanding epochs.
  • Policy computation: compute optimal linear feedback (Fm, fm) for the fitted model (solving the single‑agent LQ control under the perceived dynamics and cost).
  • Exploration: inject scaled i.i.d. noise vm_t with decaying scale αm; ϵ‑greedy style epoch structure determines exploitation vs exploration tradeoff.

• Theoretical analysis tools

  • Decomposition of learned estimate bias into (i) mis‑specification bias because opponents are unobserved and (ii) bias/noise from exploration; show exploration‑induced bias vanishes asymptotically.
  • Stability analysis of idealized best‑response dynamics around the Nash equilibrium, yielding a stability constant ζ that controls contraction.
  • Finite‑sample concentration and martingale arguments to obtain high‑probability bounds on estimation error and policy error; coupling with the dynamic (best‑response) system to derive global convergence rates.
  • Almost‑sure asymptotic convergence via standard stochastic approximation / epoch‑based decaying exploration arguments.

• Experiments

  • Simulations in duopoly dynamic Cournot model with sticky prices: sweep price stickiness s, compare learning trajectories to the complete‑information Nash, evaluate profits, total surplus, Herfindahl‑type concentration measures.
  • Variants: with/without public aggregate output release; different exploration decay schedules; a test with weakly nonlinear inverse demand to assess robustness beyond exact LQ structure.

Implications for AI Economics

• Decentralized RL in markets: Even when deployed algorithms are “radically uncoupled” (they do not observe opponents and treat the environment as single‑agent), they can converge to the competitive (complete‑information) feedback Nash equilibrium in a broad LQ class—so lack of explicit coordination does not necessarily induce persistent non‑equilibrium outcomes in these dynamics.
• Transitional welfare risks: The path to equilibrium matters. During learning, firms can suffer profit losses and, under significant market frictions (high price stickiness), total welfare can decline and market concentration can rise. Regulators and platform designers should consider transient harms from algorithmic learning, not just equilibrium properties.
• Information disclosure as a policy tool: Publicly revealing simple aggregate signals (e.g., total output) materially speeds convergence and mitigates welfare losses. Disclosure policies can be a practical lever to improve market outcomes when many automated learners adapt in parallel.
• Design implications for firms and platform operators: Learning hyperparameters (epoch growth, exploration decay) and mechanisms for limited aggregate reporting meaningfully affect both convergence speed and welfare. Firms optimizing only for own adaptation may create negative externalities during learning.
• Stability matters: The intrinsic stability of the Nash equilibrium (ζ) governs whether single‑agent optimal exploration schedules suffice to achieve fast learning. In unstable games, learning may oscillate or converge slowly—highlighting situations where intervention (information release or coordinated exploration) may be justified.
• Interpretability and identification: Even though policies converge to Nash, players do not identify true system parameters; learned models reflect aggregate strategic effects. This has implications for interpretability, auditing, and robustness of deployed ML agents in markets.

Limitations and open questions (practical relevance) - The main theory relies on the LQ structure and technical assumptions (sub‑Gaussian noise, boundedness, feasibility of perceived problems, stability condition). Generalization to highly nonlinear, high‑dimensional, or richer information structures requires further work. - While simulations show robustness to mild nonlinearity, real market environments could produce richer failure modes (multi‑equilibrium dynamics, strategic information withholding, adversarial behavior) not covered here. - Policy/design tradeoffs (how much aggregate information to disclose, how to tune exploration centrally) merit deeper investigation connecting welfare, strategic incentives, and regulatory constraints.

Summary: The paper rigorously shows that simple, uncoupled single‑agent learners can converge to full‑information Nash equilibria in LQ stochastic games, quantifies rates and the role of stability, and demonstrates substantive welfare and concentration effects in a dynamic Cournot application—while identifying information disclosure as an effective mitigation.