A safe deep reinforcement learning controller can halve simulated building heating costs while always honoring grid flexibility requests; the adaptive safety filter secures compliance but incurs a small rise in comfort breaches compared with unconstrained RL.

Main Finding

A model-free safe deep reinforcement learning (DRL) framework — combining a Deep Deterministic Policy Gradient (DDPG) controller with a Real-time Adaptive Safety Filter (RASF) — can optimize building space heating to provide demand-side flexibility while guaranteeing full compliance with operator flexibility requests. In simulation on a living-lab apartment, the approach achieves up to 50% energy/cost savings versus a rule-based controller and outperforms an unconstrained DRL agent on energy and cost metrics, at the cost of only a small increase in comfort-temperature violations.

Key Points

• Problem: Buildings are large energy consumers and can provide flexibility to the grid, but DRL controllers’ exploratory behavior risks violating contractual energy constraints or occupant comfort.
• Core contribution: A model-free RASF that enforces cumulative energy constraints in real time without requiring a physical model or prior system identification.
• DRL core: DDPG (actor-critic) handles continuous valve-opening control (heat pump). Agent state includes temperatures, solar/ambient conditions, time features, BAU energy, energy already used in the flexibility window, and flexibility window timing.
• Safety filter mechanics:
  • Compute remaining average safe action u1,t = Eremaining,t / ttoend,t where Eremaining,t = EBAU,t − Et (ensures cumulative budget).
  • Adapt tolerance τt based on normalized temperature risk and price favorability: τt = τbase,t · (w1·µtemp,t + (1−w1)·µprice,t)^1.5
  • Construct bounds umaximal,t = min(u1,t(1+τt),1) and uminimal,t = max(u1,t(1−τt),0).
  • Clip or adjust DRL action ut to usafe,t within [uminimal,t, umaximal,t]; this guarantees the cumulative flexibility constraint is met while allowing adaptive slack.
• Reward design balances thermal comfort, electricity cost, and flexibility compliance (penalty α for constraint violations).
• Results (simulation):
  • Full compliance with flexibility requests when RASF is active.
  • Up to 50% energy/cost savings relative to a rule-based controller.
  • Better energy and cost performance than a standalone DRL agent, with only slight increase in comfort violations.
• Modeling: Hybrid Physically Consistent Neural Network (PCNN) used to simulate room thermal dynamics (linear physics component + data-driven residual).

Data & Methods

• Testbed / data:
  • Simulation based on a living-lab apartment (UMAR unit) in the NEST building (Empa, Dübendorf, Switzerland). Uses real ambient/solar data and building historical data for training/simulation.
• Thermal model:
  • PCNN: Troom,t+1 = Dt+1 (physics term) + Pt+1 (NN-learned residual). Chosen for interpretability and transferability across buildings.
• DRL algorithm:
  • DDPG (continuous action) with actor µ(S|θµ) and critic Q(S,u|θQ).
  • State vector includes: solar irradiation, ambient temperature, room and neighbor temperatures, time features, heating scenario flag, time-to-start/stop of flexibility window, BAU energy EBAU,t and consumed energy Et.
  • Reward: Rt = β·Rtemp,t − δ·Rprice,t − Rflex,t, where Rtemp penalizes comfort violations, Rprice is electricity cost, and Rflex penalizes flexibility non-compliance (with different α penalties depending on violation conditions).
• Safety filter design and implementation:
  • RASF is entirely model-free (uses observed Troom,t, electricity price ζ(t), and BAU/accumulated energy estimates) and enforces cumulative energy budget over the flexibility window.
  • Tolerance τt adapts over time and with operational context (price/temperature) via normalized indicators and a base tolerance schedule.
• Baselines:
  • Rule-based controller (RBC).
  • Standalone DRL (DDPG) without the RASF.
• Evaluation metrics:
  • Energy consumption and electricity cost savings, compliance with flexibility message (cumulative energy within window), and thermal comfort violations.

Implications for AI Economics

• Monetizing Flexibility: Model-free safe DRL with guaranteed compliance enables buildings to reliably participate in flexibility markets (capacity, load-shift, or energy-constrained products) without extensive modelling costs, increasing potential revenue streams for building owners and aggregators.
• Reduced transaction/friction costs: The RASF’s model-free nature lowers setup and maintenance costs compared with MPC or model-based safety layers (less requirement for building-specific identification), improving scalability and lowering barriers for mass participation.
• Risk mitigation and contracting: Guaranteed compliance addresses counterparties’ risk concerns (grid operators/market platforms), enabling tighter contracts and potentially better prices for flexibility due to lower delivery risk.
• Market design and pricing signals: Adaptive tolerance leverages price signals to opportunistically consume when cheap and tighten during expensive periods; this suggests that well-designed price signals can be exploited safely by AI controllers to improve system efficiency.
• Aggregation & coordination: Safe, compliant single-building controllers make aggregation simpler and more reliable for virtual power plants or aggregators — reduces monitoring/penalty costs and supports trust in automated provision.
• Trade-offs & externalities:
  • Small increase in comfort violations demonstrates trade-offs between economic benefit and occupant experience; regulators/market designers may need standards or incentive alignment (e.g., comfort-compensating tariffs).
  • Dependence on accurate BAU estimates and reliable communications: errors in BAU or delays could affect service quality; markets must account for measurement and baseline uncertainties.
• Policy and regulation:
  • Demonstrated safety guarantees can support regulatory acceptance of ML-based controllers in demand-response programs, but real-world validation and certification processes will be needed.
• Future economic research directions:
  • Value stacking: quantify combined revenues from energy arbitrage, flexibility markets, and local services (ancillary) under safe DRL regimes.
  • Contract design: how to price flexibility given guaranteed delivery vs probabilistic offers.
  • Aggregator business models: optimal allocation of tolerance across portfolios to balance revenue and comfort penalties.
  • Market impacts: large-scale adoption may alter price volatility and reshape short-term markets; endogenous modeling required.

Limitations noted by the authors (economic relevance): - Results are simulation-based using PCNN; field deployment and robustness under communication delays, measurement noise, and occupant behavior variability remain to be demonstrated. - BAU estimation and data requirements for PCNN and DRL training may affect performance and baseline settlement in markets.

Overall, the paper shows a practical pathway for safe, scalable AI-enabled building participation in electricity flexibility markets — potentially lowering costs and operational risk, and expanding the supply of reliable distributed flexibility.