In a simulated 20-house Polish PV community, straightforward rule-based peer-to-peer pricing beats a Deep Q-Network pricing agent when only solar is present; adding household batteries narrows the gap and boosts savings under the RL policy, and SDR-shaped price parametrizations outperform multiplier-based rules.

Main Finding

Rule-based P2P pricing (bill‑sharing, mid‑market rate, SDR) remains highly competitive versus a centralized RL price‑setting agent in the studied residential PV community. Reinforcement learning (DQN) only outperforms the RL-free benchmarks when battery energy storage (BES) is present and when the RL agent uses an SDR‑shaped price parameterization rather than a simple multiplier scaling. Specifically, the best RL policy’s community savings rose from €734.23 (PV only) to €978.52 (PV + BES). Across learning modes, SDR‑shaped pricing dominated multiplier‑based pricing.

Key Points

• Benchmarks and RL modes
  • Rule‑based benchmarks: bill‑sharing (BS), mid‑market rate (MMR), supply–demand‑ratio (SDR) rule.
  • RL modes (centralized DQN agent):
  • RL‑M (multiplier): actions scale grid buy/sell prices (µbuy, µsell).
  • RL‑SDR‑F (fixed): single, fixed SDR‑shaped parameter pair (non‑learning control).
  • RL‑SDR‑L (learnable): agent selects discrete sensitivity pairs (α, β) that map SDR → buy/sell via a tanh mapping.
• Settlement differences matter
  • Rule‑based benchmarks settle the full internally matchable volume each hour.
  • RL posts ex‑ante buy/sell prices; internal trade only occurs if quoted prices are strictly inside the grid price corridor and the buy/sell spread is non‑negative. Thus realized traded volume can differ across mechanisms.
• Performance summary
  • In the PV‑only configuration, rule‑based benchmarks outperform the best RL policy.
  • Adding household battery storage (PV‑BES) improved RL outcomes: best RL community savings increased from €734.23 to €978.52.
  • SDR‑shaped RL (RL‑SDR‑L) consistently outperformed multiplier RL (RL‑M).
• Distributional effects
  • The distribution of benefits across households is heterogeneous; aggregate savings do not imply uniform gains.
• Practical detail: any internal spread (pp2p,buy > pp2p,sell) is retained by the market operator and counted as a cost in the community objective; the RL reward maximizes aggregate community savings, implicitly penalizing wide internal spreads.

Data & Methods

• Case study
  • Synthetic residential community: 20 heterogeneous households (8 prosumers with PV, 12 pure consumers).
  • Annual, hourly horizon (366 days); experiments use the same annual dataset for training and evaluation.
  • PV profiles from Renewables.ninja; loads from RAMP calibrated to Polish household statistics.
  • Grid prices: two‑zone TOU import price (0.249 €/kWh peak, 0.137 €/kWh off‑peak) and time‑varying export prices (market, mean ≈ 0.100 €/kWh).
  • Two technical configurations: PV‑only and PV + BES (aggregate BESS capacity 68.15 kWh, power 32.07 kW). Battery sizing/dispatch were pre‑optimized upstream by a genetic algorithm (rule‑based dispatch).
• RL formulation
  • Centralized Deep Q‑Network (DQN) agent; reward rt = Jbase_t − Jp2p_t (hourly community cost saving relative to no‑P2P reference).
  • State vector st = [sin(hour), cos(hour), discretized net balance ˜b_t (9 bins), mean SOC (5 bins), log‑transformed SDR_{t−1}].
  • Actions:
  • RL‑M: K = 8 multiplier pairs µbuy ∈ {0.2,…,0.9}, µsell ∈ {1.05,…,1.40}.
  • RL‑SDR‑F: single fixed (α,β)=(0.5,2.0) (no learning).
  • RL‑SDR‑L: K = 9 sensitivity pairs α ∈ {0.1,…,0.9}, β = 2.0.
  • SDR→price mapping (SDR‑shaped): pp2p = Cctrl · tanh(−α ln(δ + SDR^{−1})) + Cbal (buy/sell analogues), with δ = 1e‑6, Cctrl = (pbuy−psell)/2, Cbal = (pbuy+psell)/2.
  • DQN details: feedforward network 128 and 64 ReLU layers; output |A| Q values; Adam lr = 5e‑4; γ = 0.95; replay buffer 50k; batch 64; target sync every 200 steps; ε‑greedy exploration decaying 1→0.01 (λ=0.99952); 500 training episodes (each episode = one sampled day); single random seed per reported run.
• Evaluation metrics
  • Primary: community savings (Cref − Cpool).
  • Complementary: average trading price, user cost, prosumer revenue, traded energy, grid imports, self‑sufficiency index.

Implications for AI Economics

• When to expect RL value over simple rules
  • RL can improve aggregate outcomes when the system has flexible assets (here, household batteries) that increase the space for beneficial reallocation and allow adaptive pricing to exploit temporal patterns. In PV‑only settings with limited re‑dispatch flexibility, simple benchmarks can match or beat RL.
• Importance of parameterization and mechanism design
  • The RL agent’s action representation matters: an SDR‑shaped parametrization that encodes market balance semantics performed better than blind multiplier scaling. This is a reminder that embedding domain structure into action/state spaces often yields better learning outcomes and sample efficiency.
• Distributional and regulatory considerations
  • Aggregate savings can mask heterogeneous household impacts. Regulators and market designers must consider fairness and allocation rules (e.g., whether operator‑retained spreads are redistributed) — these design choices materially affect incentives and welfare distribution.
• Evaluation, robustness, and interpretability
  • Reported RL results stem from a single training seed/run; stochasticity in training and environment should be explored (multiple seeds, cross‑validation) before concluding robustness. Also, centralized RL policies are easier to evaluate but may be less transparent; rule‑based mechanisms remain attractive for interpretability and predictable distributional outcomes.
• Recommendations for future AI economics work
  • Compare RL against strong, economically meaningful benchmarks (as done here); test multiple seeds and longer evaluation samples.
  • Explore alternative objectives (e.g., social welfare, fairness‑aware rewards, operator revenue constraints).
  • Consider decentralized or multi‑agent formulations to capture strategic behavior and privacy constraints.
  • Investigate settlement rules (redistribution of spreads, budget balance) as part of the learning environment — operator revenue retention changes incentives and learned policies.
  • Study continuous‑action or policy‑gradient methods for finer price setting, and sensitivity to market price inversion events (export > import).

Limitations to note: the study uses a centralized agent with exogenous state transitions, a single training realization per mode, and a specific settlement rule where operator‑retained spreads are counted as cost; battery sizing/dispatch was optimized outside the RL loop. These modelling choices influence generalizability and should be addressed in follow‑up work.