Planner architecture changes outcomes: search-style LLM planning outperforms predefined and sequential executors in a simulated smart-grid demand-response benchmark. Enforcing event-level feasibility before quality prediction cuts selection regret sharply, showing that feasibility gating matters more than simple declared-mode agreement.

Main Finding

Architectural choice among planning strategies for LLM-based controllers materially changes both strategic and physical outcomes in a cyber-physical demand-response benchmark. A search-style executor is the oracle in baseline seeds, but selecting that executor reliably from pre-decision state is nontrivial. Simple mode agreement (declared = executed) is insufficient for preserving physical objectives (voltage/import); deterministic feasibility gating (filtering out executors that will miss event-level deadlines) substantially improves selection performance, but the remaining challenge is choosing the highest-quality option among multiple feasible executors. Model-dependent declaration collapse and heavy-tailed latency imply deployment risk and argue for probabilistic, risk-aware gates.

Key Points

• Three distinct evaluation dimensions are separated and measured: strategy heterogeneity (do different executors induce different outcomes?), execution fidelity (does following a declared mode preserve the intended objective?), and adaptive selection (can pre-decision features pick a low-regret executor?).
• In matched forced-counterfactuals (same seeds, same randomness, same prosumer draws), architecture changes outcomes: forced SEARCH was the best (oracle) across five baseline seeds.
• Mode agreement is not enough: replacing the objective (objective substitution) while keeping mode agreement yields identical declared/observed modes but increases cumulative voltage shortfall by 2.68× — showing declared execution can be faithful but physically ineffective.
• Adaptive selection from pre-decision observables is hard: a prespecified ridge regressor (stress-group out-of-fold) yields mean paired-regret 90.7 (95% bootstrap CI [73.8, 108.6]) and no clear value over always choosing SEQUENTIAL.
• Applying a deterministic feasibility filter (removing executors known to miss event-level decision deadlines) before quality prediction reduces regret to 29.0 [19.3, 40.4] and improves performance over fixed SEQUENTIAL by 61.1 [39.6, 82.2].
• When all executors are feasible (all pass deadlines), a selection model does not outperform always choosing SEARCH, localizing the remaining difficulty to within-feasible quality differentiation.
• Multimodel (five-model) interface experiments show declaration behavior is model-dependent (some declarers always collapse to one executor) and observed wall-time/declaration latency is heavy-tailed; thus feasibility and deadline compliance should be treated probabilistically.
• Methodological contributions include a physics-grounded benchmark, the abstraction of planning-induced control trajectories, and a strict counterfactual protocol (paired forced-modes, exact-prompt caching, common random numbers, critic isolation, event-level deadlines).

Data & Methods

• Task and domain: demand-response coordination on a 24-hour radial feeder with 40 heterogeneous prosumers (4 laterals × 10 nodes). Key system constraints: feeder-head import cap 1.1 MW and voltage floor 0.95 pu. Cooling-dominated load and afternoon PV create eight hours with baseline violations (e.g., uncontrolled hour 17: 1.644 MW import, min voltage 0.9344 pu).
• Executors (strategy set M): PREDEFINED (compile one allocation once), SEQUENTIAL (dispatch step, observe, rebuild suffix), HIERARCHICAL (allocate across laterals/segments), SEARCH (generate candidate schedules, evaluate by simulated power-flow, choose best). Policies are typed JSON (mode, target rule, hierarchy depth, branching, replan trigger, intensity, message theme, confidence, rationale).
• LLM boundary: LLMs only (a) declare typed policies or give bounded per-mode score shifts and (b) generate short operator messages and bounded prosumer persuasion shifts. Numerical schedule construction, prosumer base dynamics and actions, stochastic draws, and power-flow (linearised DistFlow) are implemented as explicit code.
• Prosumer response model: heterogeneous archetypes, resistance, neighbourhood memory. Base cooperation probability (reciprocity, exploitation memory, comfort fatigue, bias). For 40% of targeted prosumers, an LLM evaluates the operator message and returns a bounded willingness shift δ ∈ [−0.30, 0.30], attenuated by resistance. Final compliance and realized curtailment are stochastic agent decisions (not produced directly by the LLM).
• Objective and metrics:
  • Physical objective J = kW_over_cap + 4000 * sum(Δv) + 0.25 * kWh_curtailed (lower is better).
  • Paired routing regret per scenario q: Rρ(q) = J(ρ(q), q) − min_m J(m, q).
  • External evaluation metrics: mode agreement; faithfulness (Jaccard overlap declared vs issued targets); groundedness (share of directives ≤ 105% of true flexible load); minimality; coherence (revision churn); directive realization (realized/requested).
• Isolation and counterfactual protocol: forced-mode paired comparisons with exact-prompt caching, independent deterministic random streams but common prosumer-response draws across strategies (seed, t, node), critic isolation (LLM critic disabled in causal forced banks), event-level deadline feasibility checks (no planning event may exceed decision deadline), per-declaration/revision nominal costs.
• Experimental design overview:
  • LLM: Llama-3.3-70B-Instruct (temp 0.2, ≤400 completion tokens) for main E1–E7 runs with seeds {7,13,42,101,202} / {7,13,42} depending on experiment.
  • Experiments:
    • E1: natural full declaration vs bounded hybrid mode-advice — natural collapsed to SEQUENTIAL always; hybrid collapsed to HIERARCHICAL always.
    • E2: forced all four strategies across five seeds — isolates executor differences.
    • E3: sweep forecast noise σ ∈ {0, .05, .10, .20, .30, .40}.
    • E4: cross decision deadlines {2,4,8,30}s with deterministic deadline gate.
    • E5–E6: evaluate execution fidelity and strategic response (vary aggregator honesty; per-agent attribution of directives).
    • E7: factorial bank (4 stress groups × 3 noise levels × 2 deadlines × 2 mean resistance × 3 seeds) → 144 scenarios, 576 forced episodes; ridge models predict selection cost with stress-group holdout and bootstrap uncertainty (5000 draws). Post-hoc constraint-aware analysis removes modes that violate deadlines before predicting costs.
  • Multimodel extension (M0–M5): replay interfaces across five models (Llama-3.3-70B-Instruct, DeepSeek-V4-Pro, Gemma-3-27B-IT, GLM-5.2, MiniMax-M3) for 300 declarations and other interface-call records to probe declaration diversity, state dependence, bounded advice behavior, persuasion shifts, and observed latency (M5 reports wall time). This extension has no feeder rollouts (tests interface behavior and deployment assumptions).
• Code and data: released at the authors’ GitHub repository.

Implications for AI Economics

• Architectural choice is an economic decision with measurable expected-regret consequences. Investments in more compute-intensive executors (e.g., search) can be warranted ex post but require reliable selection policies ex ante to realize value. Paired counterfactual benchmarking reveals the true marginal value of architecture choices, which is essential for cost–benefit analysis of compute/latency trade-offs.
• Deterministic feasibility gating (filtering executors that will miss event-level deadlines) acts like a regulatory or contractive constraint that reduces selection risk and raises realized welfare — analogous to imposing ex ante enforceable constraints in market design to prevent violations. In practice, feasibility checks are cheap and often more valuable than noisy quality predictions.
• Residual selection difficulty among feasible options implies a need for richer signals or pricing mechanisms: either enrich the observable state (instrumental variables, more informative features) or price flexibility/execution options so that operators internalize selection uncertainty (option value, insurance premiums for latency/computation).
• Mode agreement as an audit signal is insufficient for economic assurance. Verifiers should evaluate groundedness and realized physical outcomes, not only declared modes; this reduces the risk of strategic substitution where stated intent diverges from effective action — relevant to contracting, accountability, and liability for AI-mediated controllers.
• Bounded LLM interfaces and explicit separation of roles (planner vs executor vs physical model) reduce moral hazard and make causal inference tractable — useful design principles for markets and platforms that coordinate autonomous agents via AI (clearly specified interfaces support better audits and contracts).
• Heavy-tailed latency and model-dependent declaration collapse imply deployment risk (congestion, tail latency) that affects real-time markets and safety margins; regulators and platform designers should treat feasibility probabilistically and price or reserve capacity to hedge tail risks.
• The counterfactual, physics-grounded protocol offers a template for valuing architectural upgrades: use paired, common-random-number comparisons to estimate marginal gains from strategy changes before committing capital to compute or model upgrades. This is directly applicable to procurement decisions and investment appraisal in AI-enabled infrastructure.