A standardized bridge from economics to reinforcement learning: the paper prescribes how preferences, frictions and information should map to reward, discounting and learning designs so RL-based economic models become interpretable, comparable and reproducible. Simulations show that changing frictions, risk attitudes and time preferences systematically alters learned policies and their divergence from static equilibria.

Main Finding

The paper provides a systematic methodological architecture that maps standard economic ingredients (preferences, frictions, information, time horizons) to Reinforcement Learning (RL) components (reward, constraints/environment, observation model, discount factor, exploration). It frames equilibrium outcomes as benchmarks for evaluation rather than axioms, enabling interpretable, comparable, and reproducible use of Q‑learning and related RL methods in applied economics. A simulation shows how varying frictions, risk attitudes, and intertemporal preferences changes learned policies, their stability, and their divergence from static equilibria.

Key Points

• Proposes a general specification (an “architecture”) linking economic theory to RL elements so models are comparable across studies.
• Emphasizes that the contribution is methodological: specifying how to encode economic structure into learning environments rather than inventing new RL algorithms.
• Treats economic equilibrium as a benchmark for evaluation; deviations are measurable diagnostics, not modeling failures to be suppressed.
• Organizes models along explicit dimensions:
  • behavioural preferences (risk, time preferences)
  • institutional frictions (constraints, adjustment costs)
  • economic environment class (market structure, number/type of agents)
  • information structure (observability, signal/noise)
  • learning & exploration mechanisms (algorithms, exploration schedules)
  • evaluation metrics (convergence, welfare, distance to equilibrium, robustness)
• Demonstrates via simulation that policy outcomes, convergence properties, and welfare comparisons systematically vary with frictions and preference parameters.

Data & Methods

• Methodological framework: specifies mapping rules from economic building blocks to RL design choices, e.g.:
  • preferences → reward function (utility specification, risk-sensitive transforms)
  • time horizons/intertemporal preferences → discount factor and episode length
  • frictions → state/action constraints, transition dynamics, penalty terms
  • information structure → observation space, partial observability, belief/state representations
  • institutional rules/multi-agent settings → environment architecture and interaction protocol
  • learning/exploration → choice of Q‑learning (or related) algorithm, exploration schedule, function approximation
• Evaluation protocol: recommends a battery of diagnostics (examples consistent with the paper’s goals):
  • convergence behavior (learning curves, stability of policies)
  • policy distance to equilibrium (normed differences in actions or allocations)
  • welfare/regret comparisons (expected utility loss relative to benchmark)
  • robustness checks (sensitivity to seeds, hyperparameters, alternative information structures)
  • dynamic performance (stability, cycles, path-dependence)
• Simulation study: implements the architecture on an illustrative environment and varies frictions, risk attitudes, and discounting to show how learned policies and their relation to static equilibria change. Uses these outcomes to demonstrate interpretability and diagnosability of deviations.

Implications for AI Economics

• Interpretability: makes clear how economic assumptions are encoded in RL experiments, aiding economic interpretation of learned behavior.
• Comparability: by standardizing specification dimensions and evaluation metrics, different studies can be compared and cumulated.
• Reproducibility: explicit mappings and recommended diagnostics reduce ambiguity in how RL studies implement economic models.
• Theory–learning dialogue: reframes deviations from equilibrium as informative signals that can refine theory (e.g., reveal when frictions or information assumptions matter).
• Policy & calibration: provides a structured way to use RL-derived policies for counterfactual analysis and policy evaluation while clearly documenting assumptions.
• Research practices: motivates publishing full specification (reward, discount, observability, friction encoding), evaluation code, and standard benchmark tasks for economic applications of RL.
• Limitations and next steps (implied): need for community standards, computational costs of large-scale simulations, sensitivity to algorithmic choices and function approximation—suggests further work on shared benchmarks and robustness toolkits.

Adopting this architecture should make RL applications in economics more transparent, comparable, and useful for both positive and normative analysis.