Learning-based advisers can deliver more information than static theory predicts: with aligned incentives, reward-driven adaptation turns noisy policies into informative advice; with conflicting incentives, continual adaptation prevents equilibrium but sustains cyclic, highly informative communication that raises payoffs above any static outcome.

Main Finding

Reinforcement-learning (tabular Q‑learning with softmax exploration) used by an advisor in a Crawford–Sobel cheap‑talk environment produces robustly informative communication through a simple reward‑driven adaptation rule. When sender and receiver preferences are aligned, learning typically converges to highly informative (though generally not fully revealing) messaging policies with welfare bounded well above the babbling benchmark (a tight lower bound; in the paper’s baseline calibration ≈98% of full‑revelation surplus). When preferences are misaligned, no stationary fixed point is stable—learning induces persistent cycles. Those cycles sustain very informative communication and can yield payoffs strictly larger than any static (partitional) equilibrium of the stage game.

Key Points

• Model setup
  • Discrete Crawford–Sobel environment: state X uniformly on a K‑point grid in [0,1]; sender observes X and sends message m; receiver chooses action y.
  • Quadratic payoffs: sender U_S = −(y − x − b)^2 (bias b ≥ 0); receiver U_R = −(y − x)^2.
• Algorithmic sender
  • Sender is a tabular Q‑learner (softmax/Boltzmann action selection with persistent exploration ε>0 and temperature τ>0).
  • Payoff noise η (zero mean, bounded variance) is allowed; step‑size conditions satisfy standard RL assumptions (Σα_t = ∞, Σα_t^2 < ∞).
• Receiver
  • Short‑lived, sophisticated best‑responders who know the sender’s current policy μ_t and play y(m) = E[X | m, μ_t]. They are myopic and do not try to influence sender learning.
• Reward‑driven adaptation
  • The Q‑learning update reinforces messages that yielded higher sender payoffs and attenuates others; this selection force channels the policy evolution toward informative mappings even from uninformative initial policies.
• Aligned preferences (b = 0)
  • Any attracting (stable) learned policy must transmit substantial payoff‑relevant information.
  • Stability results give a tight lower bound on welfare relative to full revelation; in the baseline parameters this bound is ≈98%.
  • Persistent exploration prevents full revelation: experimentation makes the receiver’s posterior less certain, which in turn induces the sender to exaggerate or pool, blocking convergence to the fully revealing equilibrium.
• Misaligned preferences (b > 0)
  • No stationary learned policy is stable under the dynamics; classical partitional equilibria are not attracting.
  • Typical learned pattern (from simulations): a threshold partition where states above a threshold are pooled (one message for many high states), while states below the threshold are almost fully revealed but the policy continually cycles because of incentives to inflate messages (positive bias). The resulting dynamics are non‑convergent but highly informative.
  • These dynamic cycles can yield sender and receiver welfare that exceed the best static equilibrium payoffs from Crawford and Sobel.
• Robustness and scope
  • Results hold under modest generalizations (e.g., nonzero discount β, alternative softmax vs ε‑greedy variants) and are illustrated/supported by simulations.
  • Authors acknowledge external‑validity limits given focus on tabular Q‑learning, but argue the mechanism (reward‑driven selection under exploration) plausibly carries over to richer RL architectures.

Data & Methods

• Analytical approach
  • Embed Q‑learning sender into the cheap‑talk game; approximate discrete Q‑learning dynamics by an ODE as learning/temperature/exploration parameters evolve under standard assumptions.
  • Study limit points (attracting policies) of the induced policy dynamics and characterize when they must be informative or when no fixed points exist.
  • Derive welfare bounds for attracting policies in the no‑bias case.
• Computational approach
  • Simulations explore dynamics across parameter values (K, τ, ε, learning rates, bias b, small reward noise η, discount β) to illustrate typical learned languages and welfare outcomes.
  • Simulations confirm: (i) learning from uninformative starts leads to informative languages in aligned case; (ii) exploration rates that are too large prevent full revelation; (iii) misaligned case yields persistent cycles and supra‑equilibrium payoffs.
• Key modeling assumptions to note
  • Sender: tabular Q‑learning with softmax/ε‑exploration; exploration and temperature remain bounded away from zero in the limit (τ>0, ε>0) for main analysis (but some results extend when they vanish suitably).
  • Receiver: knows sender policy μ_t and best‑responds each period; receiver is myopic and does not strategically influence learning.
  • Payoff feedback observed with additive noise.

Implications for AI Economics

• Algorithmic advice can improve information transmission relative to static strategic equilibria
  • Even imperfect RL algorithms can produce substantially informative communication; in some misaligned settings, dynamics even outperform static equilibria in welfare terms.
• Exploration has normative costs in communication contexts
  • Persistent exploration (needed for learning) reduces receiver confidence in messages, inducing caution and encouraging sender exaggeration or pooling—so exploration policies should be designed carefully when the objective includes truthful information transmission.
• Alignment matters but is not a panacea
  • Alignment (b ≈ 0) yields high welfare under learning, but full welfare maximization may still fail because of exploration and reward‑driven adaptation. Designers cannot assume model‑free RL will automatically implement fully truthful revelation even when interests align.
• Dynamic behavior can be socially beneficial yet unpredictable
  • When preferences diverge, RL may generate non‑stationary but highly informative communication patterns that deliver higher payoffs than static equilibria; however, predictability and stability are lost, which may be problematic for human trust and interpretability.
• Design and regulatory considerations
  • Platform designers should (a) minimize unnecessary exploration visible to users or signal when exploration is occurring, (b) account for how users will infer algorithmic exploration from realized outcomes, and (c) consider mechanisms that allow receivers to “teach” or credibly influence the algorithm (the paper assumes myopic receivers).
  • Regulators and auditors should be aware that algorithmic recommenders may intentionally or unintentionally distort communicated information because of reward‑driven learning dynamics—this can lead to welfare gains or losses depending on alignment and exploration.
• Broader theoretical relevance
  • Connects RL learning dynamics to the economics of cheap talk, evolutionary language emergence, and multiagent learning: reward‑driven adaptation is a simple selection mechanism producing informative languages but also producing systematic frictions (cycling, pooling) absent under full rationality.

Limitations and open questions - Tabular Q‑learning is a convenient, tractable benchmark; how the mechanisms extend to deep RL, function approximation, contextual bandits, or settings where receivers try to teach is left for future work. - The assumption that receivers know the sender’s policy each period and are myopic is strong; outcomes may differ when receivers are strategic and patient.