← Papers

An entropy-regularised RL algorithm (SAC) learned inverter-like continuous control that eliminates compressor starts and cuts thermal discomfort by up to 90.7% for a small cost premium, whereas PPO collapsed to aggressive on–off cycling that worsens wear and under-heats; results come from surrogate-trained agents validated on the BOPTEST emulator.

Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

Faizan Ahmed, Aniket Dixit, James Brusey · August 10, 2026

arxiv quasi_experimental medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 11, 2026

Faizan Ahmed unresolved corpus identity
Aniket Dixit unresolved corpus identity
James Brusey unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

Faizan Ahmed provider ID
Aniket Dixit provider ID
James Brusey provider ID

With a monetised compressor-wear term in the reward, Soft Actor–Critic learned an inverter-style continuous-modulation policy that eliminated compressor starts and greatly reduced thermal discomfort at a modest cost increase, while PPO converged to bang–bang cycling that increased wear and under-delivered heat.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.

Summary

Main Finding

Adding a monetised compressor-wear term to a heat-pump RL reward makes switching a first-class economic objective, and the choice of RL algorithm strongly determines whether the learned controller minimizes wear. With the same reward and observations, Soft Actor–Critic (SAC) learns a continuous, inverter-style modulation policy that eliminates compressor starts (0 starts/day) and greatly reduces thermal discomfort (up to 90.7% reduction) for a modest cost increase, while Proximal Policy Optimisation (PPO) collapses to bang–bang control that cycles more than the baseline.

Key Points

Reward design: the per-step reward = −(thermal-discomfort + electricity-cost + monetised-wear). Wear is levelised to a per-start cost c_cyc = €0.0133 (amortised replacement over rated starts). All weights set to 1 so wear is directly comparable to energy cost.
Two zero-wear optima: never run (ruled out by discomfort) or run continuously (inverter-like). Whether RL finds the latter depends on the algorithm.
Algorithmic divergence:
- SAC (off-policy, maximum-entropy) biases towards interior, stochastic actions; converged policy modulates capacity continuously, never drops off → zero starts/day.
- PPO (on-policy, no entropy floor) biases toward endpoint mass → bang–bang cycling; cycles more than baseline.
Quantitative outcomes (14-day BOPTEST scenarios, peak and typical):
- Baseline starts/day: 1.07 (peak) and 1.50 (typical).
- SAC starts/day: 0.00 (both scenarios).
- PPO starts/day: 4.29 (peak) and 2.64 (typical) — more than baseline.
- Peak-day KPIs: SAC reduced thermal discomfort from 8.382 to 0.777 K·h (−90.7%) for an 11.5% operating cost increase (cost €0.909 → €1.013).
- SAC draws ~11% more source energy than baseline; the cost premium buys more delivered heat rather than improved tariff timing.
Practical observation: SAC’s modulation range stayed inside (ε, 1] and never triggered the binary “off” threshold (ε = 1e−3), i.e., the compressor stays engaged and modulates; PPO spends a large share of steps fully off (60.6% peak, 82.3% typical).
Limitations noted by authors: permissive duty threshold (ε), small absolute per-start monetary value, training on a SINDy surrogate (though validated on BOPTEST emulator), and the need to test stricter minimum modulation constraints.

Data & Methods

Control formulation:
- Discrete-time MDP, timestep Δt = 900 s (15 min), episodes = 14 days (1344 steps).
- Action: u_HP ∈ [0,1] (normalized compressor capacity).
- Observation: zone temp, ambient, irradiance, internal gains, setpoints, price, time-of-day encoding, plus 8-step look-ahead forecasts; when wear term active, includes previous on/off and dwell time.
Reward details:
- Thermal-discomfort term: integrated band violation in K·h.
- Electricity-cost: P_tot Δt π_t /1000 (euros).
- Wear: J_wear,t = c_cyc · max(d_t − d_{t−1}, 0) with d_t = 1[u_HP,t > ε], c_cyc = €0.0133 per start; short-cycle surcharge applied for starts separated by <4 steps.
Surrogate plant and training:
- Agents trained on a SINDy-identified surrogate (fast, data-driven) to collect 5×10^6 steps affordably.
- Companion map supplies electrical power for cost/wear terms.
- Algorithms: SAC and PPO implemented with Stable-Baselines3; same NN sizes, γ = 0.99, lr = 3e−4; only algorithm differs.
- Each agent trained 5×10^6 steps, then evaluated out-of-sample on the BOPTEST emulator (two scenarios: peak and typical heating periods under dynamic tariff).
Evaluation metrics:
- BOPTEST closed-loop KPIs (thermal discomfort, cost, energy, emissions).
- Cycling metrics: switches/day, starts/day, monetised wear (€/day), mean step-to-step variation (TV).
- Command statistics (range, fraction of steps below duty threshold).

Implications for AI Economics

Algorithm choice is an economic design lever: off-policy, entropy-regularised methods (SAC) can implicitly enforce hardware-friendly smoothness and thereby reduce life-cycle costs; on-policy approaches without entropy bias (PPO) can produce policies that minimize short-run operating cost at the expense of greater wear and higher long-term capital replacement risk.
Monetising hardware degradation aligns RL incentives with lifecycle economics: expressing per-start wear in the same currency as operating cost lets the agent trade off immediate energy cost vs cumulative equipment wear. This makes a single scalar reward weight (w_wear) a transparent policy lever on the cost-vs-wear Pareto frontier.
Small per-action externalities can flip design choices: although per-start cost here is small (€0.0133/start), aggregated starts and long-term replacement make the economic effect material. The paper’s back-of-envelope: SAC avoided 15 baseline starts in one scenario (worth €0.20), exceeding its €0.10 operating premium → a >2× return on the premium; break-even per-start ≈ €0.007 in that experiment.
Policy evaluation and procurement should include hardware-wear KPIs: evaluations that report only energy and comfort can miss negative externalities. For commercial/regulated deployment, require start-count or levelised-wear metrics in controller certification and vendor contracts.
Market and regulatory design:
- Utilities and standards bodies could incentivise modulation-friendly operation (e.g., tariff structures or rebates that reflect lifecycle savings), or mandate wear-aware control in performance specifications.
- For single-stage (on/off) units, enforcing minimum-runtime or other constraints in controllers may be necessary; for inverter-capable equipment, algorithmic approaches like SAC can learn modulation without explicit engineering.
Cautions for economic modelling and deployment:
- Algorithmic regularisation is not a substitute for careful modelling of device physics and operational constraints (e.g., minimum stable modulation band): reward-based monetisation should be validated against realistic device limits.
- Surrogate-trained RL must be validated on high-fidelity emulators and field trials before assuming projected life-cycle savings.
- Sensitivity analyses are required: per-start cost, duty threshold, and wear-weight w_wear materially change the Pareto frontier; these are policy knobs for economists/policymakers.
Recommendations for practitioners and economists:
- When designing RL-based controllers for capital-intensive equipment, include levelised component-wear costs in the objective (monetise replacement amortised over expected lifetime operations).
- Prefer entropy-regularised, off-policy algorithms (or explicit smoothness regularisers) when hardware longevity and smooth control are priorities.
- Report and optimise for start-count or equivalent wear KPIs in addition to energy and comfort.
- Run sensitivity sweeps on the wear price and minimum-modulation thresholds; benchmark RL policies against MPC and deployed heuristics on both energy and wear KPIs.

Overall, the paper illustrates that algorithmic architecture and reward monetisation are economically consequential design choices: they change not only short-run operating cost and comfort trade-offs, but also long-run capital wear and replacement economics.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The study reports consistent, large differences between two RL algorithms and validates learned policies on a standard high-fidelity emulator (BOPTEST), giving good internal evidence; however all training is in simulation (a surrogate model identified offline), only one heat-pump test case and two scenarios are used, and no hardware or field validation or broader baselines (e.g. MPC) are provided, limiting external validity. Methods Rigormedium — Methodology is clear and reproducible: identical MDP/reward/observations, long training budget on a documented SINDy surrogate, evaluation on a community emulator, and sensible KPIs; but key weaknesses include reliance on an externally identified surrogate (identification left outside scope), a permissive duty threshold that favors the 'always-on' optimum, a single test-case, fixed wear weighting, and lack of additional algorithmic and MPC baselines or hardware experiments. SampleAgents trained for 5×10^6 steps on a data-driven SINDy surrogate of the BOPTEST bestest hydronic heat pump; policies then evaluated on the genuine BOPTEST emulator for two 14-day scenarios (peak and typical heating periods) under a highly dynamic tariff; observations include zone and ambient temps, irradiance, internal gains, setpoints, current price, time-of-day encodings and 8-step exogenous forecasts; action is continuous normalized compressor capacity uHP∈[0,1]; metrics: starts/switches, monetised wear, thermal discomfort (K·h), cost, energy, emissions. Themesinnovation adoption IdentificationHold environment, reward, observations and hyperparameters constant while varying only the RL algorithm (SAC vs PPO); agents are trained on an offline SINDy surrogate (5e6 steps) and evaluated out-of-sample on the BOPTEST bestest hydronic heat-pump emulator across two 14-day scenarios, attributing differences in cycling/comfort/cost outcomes causally to algorithm class. GeneralizabilityTrained primarily in simulation using a SINDy surrogate—transfer to real hardware not demonstrated, Single BOPTEST hydronic heat-pump test case (one-zone) limits applicability to different building types and system architectures, Permissive duty threshold (ε=1e-3) may overstate feasibility of continuous 'always-on' operation for real inverters with higher minimum modulation limits, Single fixed wear monetisation and equal weighting (wdisc=wcost=wwear) — results may change with different economic parameters, Only two RL algorithms tested and no MPC or hardware controller baselines, 15-minute timestep and episode length choices may not match other operational regimes

Claims (8)

Claim	Direction	Outcome	Confidence & Evidence	Details
When trained with the same environment, reward, and observations, SAC learned a continuous-modulation heat-pump policy with zero compressor starts per day in both evaluated scenarios, whereas PPO learned a bang-bang policy that cycled more than the baseline. Other	mixed	Compressor starts and switching frequency	Reading fidelity high Study strength medium	n=2 SAC: 0.00 starts/day; PPO: 4.29 starts/day in peak and 2.64 starts/day in typical, versus baseline values of 1.07 and 1.50 0.48
On the peak heating scenario, SAC reduced thermal discomfort by 90.7% relative to the baseline while increasing operating cost by 11.5%. Consumer Welfare	mixed	Thermal discomfort and operating cost	Reading fidelity high Study strength medium	n=1 90.7% reduction in thermal discomfort; 11.5% increase in operating cost 0.48
On the typical heating scenario, SAC reduced thermal discomfort by 33.1% relative to the baseline, but increased operating cost from €0.413 to €0.631. Consumer Welfare	mixed	Thermal discomfort and operating cost	Reading fidelity high Study strength medium	n=1 33.1% reduction in thermal discomfort; €0.218 increase in operating cost 0.48
SAC eliminated the baseline compressor cycling in both scenarios, recording zero switches, zero starts per day, and zero monetized wear, while the baseline recorded 1.07–1.50 starts per day. Other	positive	Compressor switching, starts per day, and monetized compressor wear	Reading fidelity high Study strength medium	n=2 0.00 starts/day and €0.0000/day wear for SAC versus 1.07–1.50 starts/day and €0.0143–€0.0200/day wear for baseline 0.48
PPO increased compressor cycling relative to the baseline, producing 4.29 starts per day in the peak scenario and 2.64 starts per day in the typical scenario. Other	negative	Compressor starts per day	Reading fidelity high Study strength medium	n=2 PPO cycled 2 to 4 times more often than baseline 0.48
SAC's commanded compressor capacity remained continuously nonzero rather than switching off: it ranged from 0.007 to 0.93 in the peak scenario and from 0.001 to 0.85 in the typical scenario, with 0.0% of steps below the duty threshold in both cases. Other	positive	Continuity and modulation of compressor-capacity commands	Reading fidelity high Study strength medium	n=2 0.0% of steps below the duty threshold in both scenarios 0.48
PPO used bang-bang control, spending 60.6% of steps fully off in the peak scenario and 82.3% fully off in the typical scenario, while saturating at the full-capacity rail. Other	negative	Fraction of control steps at off/full-capacity command levels	Reading fidelity high Study strength medium	n=2 60.6% off steps in peak; 82.3% off steps in typical 0.48
PPO's lower energy cost was associated with under-delivery: it reduced peak-scenario cost by 30.6% relative to baseline but only reduced discomfort by 17.9%, and it increased typical-scenario cost savings while leaving discomfort nearly unchanged. Consumer Welfare	mixed	Operating cost and thermal discomfort	Reading fidelity medium Study strength medium	n=2 17.9% discomfort reduction in peak and 3.8% reduction in typical; 30.6% and 38.3% cost reductions, respectively 0.29