Revenue-focused RL can hit business KPIs while breaking operational rules; a trace-based 'discipline stability' evaluation shows behavior cloning or trace-prior corrections better preserve pricing and bidding discipline in simulated hotel and bidding benchmarks.

Main Finding

Outcome-only evaluation (matching a scalar KPI) can certify economically unsafe agents: policies that hit revenue targets (RevPAR) can nonetheless violate the behavioral discipline of the benchmark (price/bid distributions, pacing). Under hidden competitor state, preserving the benchmark requires trace-based evaluation and training that respects the posterior predictive distribution over benchmark actions. In a two-hotel pricing benchmark, reward-only RL (PPO variants) achieves RevPAR but fails to match benchmark trace structure; a Trace-Prior approach (and, in the symmetric case, behavior cloning) preserves price/bid distributions and pacing while allowing bounded adaptation under capacity asymmetry.

Key Points

• Problem framed: strategic economic agents are often judged by compressed KPIs (e.g., RevPAR). That is necessary but not sufficient for safe/deployable behavior because the same KPI can be produced by qualitatively different policies (undercutting, occupancy grabbing, collapsed modal prices, etc.).
• Discipline stability (defined empirically): a learned policy is discipline-stable relative to a benchmark if it preserves both the scalar outcome and the benchmark's trace distributions (action distributions, occupancy/ADR, state-sliced behavior) under the deployable information regime.
• Hidden-state aliasing is the core mechanism: the same observable state for the learner (Hotel A) can correspond to multiple competitor inventories and therefore multiple valid benchmark actions. The correct target is the posterior predictive distribution over benchmark actions, not a single action label.
• Reward-only baselines (PPO, recurrent PPO, CTDE PPO) optimize RevPAR but fail to reproduce benchmark trace distributions (price buckets, ADR/occupancy combinations). Deterministic argmax copying can increase step-level accuracy but collapses the mixture and distorts aggregate trace.
• Trace-Prior RL: estimate a full-distribution market prior pi_M(a | o_T) from benchmark traces, then train a teacher policy that preserves that distribution (e.g., via KL/regularizer) while optimizing the agent objective. This preserves uncertainty induced by hidden state.
• Behavior cloning (BC) is nearly sufficient in the symmetric two-hotel setting: BC-only stochastic sampling from the learned market prior matched the benchmark trace and RevPAR almost as well as Trace-Prior RL. But when Hotel A’s capacity differs (capacity-asymmetric stress test), Trace-Prior RL allowed bounded, economically meaningful adaptation while keeping close trace alignment; BC-only is more conservative.
• Paradigm contributions (evaluation protocol): (1) define benchmark discipline; (2) define deployable observation regime; (3) induce trace diagnostics; (4) diagnose with ablations before repairing (reward-only, memory, hidden-state oracle, argmax, full-distribution prior); (5) test persistence via student transfer and frozen deployment.
• Replication: the same discipline-stability paradigm reproduces in a second compact POMDP (hidden-budget bidding), demonstrating generality beyond the hotel-pricing toy.

Data & Methods

• Environment: finite-horizon selling POMDP, default H = 30, capacity Q (typically 100). Discrete price action set A = {100,120,140,160,180,200,220}. Guests choose Hotel A, Hotel B (Fixed RM), or outside option. Hotel B uses a deterministic FixedRM rule (depends on time, its remaining inventory q_B, market condition m_t); q_B and the rule are hidden to Hotel A in deployable regimes.
• Observation regimes:
  • NC: own state plus own corrected price history (no competitor prices/history).
  • CA: own state plus lagged market/Hotel B prices (last 3 prices). In both regimes Hotel B’s current inventory and pricing rule remain hidden.
• Metrics:
  • Outcome: RevPAR (per-available-room revenue).
  • Business decomposition: occupancy, ADR.
  • Trace diagnostics: price-bucket distributions, L1 distance and Jensen–Shannon divergence between action distributions, state-sliced L1/JS to detect local failures.
  • Transfer/interaction: student vs teacher trace distance; frozen deployment tests (NC-vs-NC, CA-vs-CA, NC-vs-CA).
• Algorithms / baselines:
  • Reward-only: PPO, recurrent PPO (R-PPO), CTDE PPO (critic sees hidden competitor info during training).
  • Trace-access baselines: BC-only stochastic copy (supervised market-price model, sample actions), BC warm-start + PPO, PPO + BC auxiliary (sampled-action cross-entropy auxiliary loss with tuned alpha), Trace-Prior teacher (full-distribution prior + RL), corrected-history Student (reduces direct market-price dependence).
• Ablation ladder: tests whether failure originates from optimizer weakness, missing memory, hidden-state ambiguity (oracle q_B predictor), deterministic collapse (argmax predictor), or repair form (sampled-action loss vs full-distribution prior).
• Key experimental findings (high-level numbers):
  • Reward-only PPO family: RevPAR ≈ 93–95 for Hotel A, with large L1/JS gaps to the benchmark price distribution.
  • BC-only stochastic copy and Trace‑Prior teacher: RevPAR ≈ 107–108 for Hotel A and nearly identical price-bucket distributions to Hotel B (L1 ≈ 0.015, JS ≈ ~0).
  • Capacity-asymmetric stress test (Q_A = 120, Q_B = 100): Trace‑Prior RL produced a small, positive RevPAR gain over BC-only (+0.764 paired mean; 95% CI [+0.125, +1.403]) while changing the A-B L1 only slightly (+0.0013), indicating bounded adaptation with preserved trace shape.
  • Hidden-state diagnostic: training a predictor for Hotel B price with oracle q_B sharply reduces label uncertainty—supporting the causal story that hidden competitor inventory drives the mixture.
• Seeds and stability: experiments run across multiple seeds (typically 5–10) and paired seed comparisons reported with confidence intervals where appropriate.

Implications for AI Economics

• Evaluation design: For economic/strategic AI systems, scalar KPI success is insufficient. Evaluators must measure trace-level behavior (action distributions, pacing, state-sliced diagnostics) under the actual deployable information regime to detect reward-hacking or discipline failure.
• Partial observability and hidden competitor/state are common in markets and multi-agent settings. When the learner cannot observe competitor private state, the training target should be the posterior predictive distribution over benchmark actions—not a single deterministic label.
• Uncertainty preservation: deterministic copying or argmax supervision can collapse uncertainty and distort macro-level behavior. Preserving stochasticity (sampled BC or full-distribution priors) is important to maintain market-like mixtures and avoid destabilizing downstream interactions.
• Practical repair toolbox:
  • Use supervised distributional priors (pi_M) estimated from benchmark traces as an intermediate structure.
  • Behavior cloning can suffice for near-perfect imitation in symmetric contexts; RL with a trace prior is useful when adaptation (e.g., different capacity/objectives) is desirable but should be constrained to preserve trace.
  • Include KL- or regularizer-based constraints to keep learned policies close to benchmark distributions while allowing bounded optimization.
• Deployment safety: include persistence/transfer tests (student models, frozen deployment, cross-regime interactions) to ensure that discipline is not contingent on availability of benchmark signals at training time.
• Policy certification: regulators or platform owners should require discipline-stability checks (the paper's minimal reporting checklist) for automated economic agents that will interact with real markets.
• Research agenda: extend discipline-stability evaluation to richer, more realistic market simulators and more diverse MARL methods; study long-term interaction effects (online co-learning) and how preserved trace structure affects market welfare and stability.
• Caveats: results come from compact, synthetic benchmarks; the paper does not claim universal failure of modern MARL methods nor that Trace‑Prior RL is the sole remedy. The recommendation is methodological: adopt trace-based evaluation and training targets that respect deployable information regimes.

If you want, I can (a) extract the paper’s ablation table and key metric numbers into a compact table, (b) draft a short checklist template you could use to audit a market-facing agent, or (c) outline an experiment plan to apply this evaluation paradigm to a larger, more realistic market simulator. Which would be most useful?