Trade-R1 curbs RL's reward-hacking in financial decision tasks by validating reasoning across retrieved evidence, reasoning chains and decisions; its dynamic semantic-reward variant generalizes better across country markets while preserving the strongest reasoning consistency.

Main Finding

Trade-R1 introduces a process-level verification framework that connects verifiable but noisy market rewards to LLM reasoning via a Retrieval-Augmented Generation (RAG) judge and a triangular consistency metric. Coupling semantic alignment with market returns (Dynamic-effect Semantic Reward, DSR) reduces reward hacking by suppressing noisy signals and amplifying valid signals, yielding better reasoning consistency and improved cross-market generalization compared with outcome-only RL and a fixed-additive semantic reward (FSR).

Key Points

• Problem: Standard RL that optimizes realized market returns in finance is vulnerable to reward hacking—policies exploit noisy historical winners and hallucinate rationales—because market returns are stochastic (r = r* + ξ).
• Core idea: Gate market rewards by a process-level semantic verification score s computed from evidence, reasoning, and decision, so only outputs with grounded reasoning receive full credit.
• Triangular consistency metric: Decomposes semantic verification into three pairwise scores (Evidence↔Reasoning factuality, Reasoning↔Decision deduction, Evidence↔Decision consistency) and averages them to produce s ∈ [0,1].
• Two semantic-reward strategies:
  • Fixed-effect Semantic Reward (FSR): G(r,s) = r + 2·s (additive, constant incentive).
  • Dynamic-effect Semantic Reward (DSR): G(r,s) = r·(0.5 + s) if r>0, else r·(2 − s). DSR multiplicatively couples alignment with return magnitude.
• Theoretical effect of DSR: For noise-driven positive returns with poor reasoning (s→0), DSR scales reward by 0.5 (variance factor ≈0.25), suppressing noisy gradient updates; for valid reasoning (s→1), it amplifies positive signals by 1.5 — improving SNR and reducing reinforcement of spurious correlations.
• Two-stage verification (RAG): retrieve concise evidence chunks (reduces eval context from ~30K to ~10K tokens) then have an LLM judge compute the triangular scores; halves evaluation time and improves reliability.
• Training and optimization: GRPO (grouped rollout policy optimization) with group-wise normalization to remove non-stationary market trend; policy model Qwen3-8B-Instruct fine-tuned end-to-end; embeddings via BGE-M3; judge model Doubao-seed-1.8.
• Data augmentation: each daily context is paired with M=15 investment beliefs (styles); ensemble-averaged outputs for evaluation; reward is 10-day forward excess return.

Data & Methods

• Datasets:
  • A-share (CN): daily Chinese financial news summaries, training Jul 2024–Jun 2025, test Jul–Oct 2025.
  • US market: daily English summaries, used for OOD testing (Jul–Oct 2025).
  • Each day produces 15 samples via belief augmentation (styles such as dividend-focused, sector-rotation, etc.).
• Model & training:
  • Policy: Qwen3-8B-Instruct, fully fine-tuned.
  • Optimization: GRPO, batch size 32, lr 1e-6, 10% warmup, sampling temperature 1.0, G=8 rollouts per query.
  • Retrieval: BGE-M3 embeddings, top-k chunk selection after mention localization.
  • Judge: Doubao-seed-1.8 computing three similarity scores; final s = mean(SE↔c, Sc↔d, SE↔d).
• Evaluation metrics:
  • Financial utility: cumulative return, Sharpe ratio, max drawdown (MDD).
  • Reasoning quality: semantic similarity score (Sim.), hallucination rate.
• Key quantitative outcomes (means over 5 seeds; A-share training):
  • A-share test (Jul–Oct 2025):
    • Market-only: Cum Ret 37.62%, Sim 0.4369, Halluc. 0.2254
    • FSR: Cum Ret 39.38% (highest), Sim 0.9560, Halluc. 0.0039
    • DSR: Cum Ret 37.76%, Sim 0.9744 (highest sim.), Halluc. 0.0012 (lowest)
  • US test (OOD):
    • Market-only: Cum Ret 12.63%, Sim 0.6586
    • FSR: Cum Ret 11.40%, Sim 0.7579
    • DSR: Cum Ret 15.34% (best OOD return), Sim 0.7768
• Ablations: two-stage RAG verification reduced evaluation context from ~30K to ~10K tokens and halved per-step evaluation time; group normalization stabilizes learning under nonstationary markets.

Implications for AI Economics

• Reward design in stochastic economic environments: Trade-R1 provides a concrete mechanism to mitigate Goodhart-style reward hacking by linking outcome rewards to process-level verification. This shows that reward functions can and should incorporate validity checks on agent reasoning to improve economic decision robustness.
• Improved SNR through semantic gating: The DSR multiplicative coupling both penalizes spurious gains and rewards logically grounded gains, changing the optimization landscape (lower variance on noisy samples, amplified genuine signals). This is a practical strategy for RL in domains with high outcome noise (finance, macro policy simulation, clinical trials, supply chains).
• Cross-market generalization: Models trained with process verification (especially DSR) generalize better OOD (trained on A-share, DSR achieved superior returns on US test set), suggesting process-level constraints reduce overfitting to historical market idiosyncrasies—important for deploying AI agents across diverse economic regimes.
• Measuring and monetizing reasoning quality: Triangular similarity and hallucination rate deliver operational metrics linking explainability to economic performance; such measures enable an economic valuation of “trustworthy” reasoning (e.g., pricing lower capital allocation to strategies with low semantic scores).
• Practical costs and operational trade-offs: Retrieval + judge overhead increases compute and latency relative to market-only RL. The paper reports ~2× evaluation speedup via RAG vs full-context; still, running retrieval + judge in production requires cost-benefit analysis (compute costs vs reduced tail risk and improved robustness).
• Broader policy and governance: Process-level verification offers auditability and traceability (evidence ↔ reasoning ↔ decision), which aligns with regulatory priorities for explainable automated decision-making in finance. It may facilitate compliance, but also raises questions about the choice of judge models and their biases.
• Research directions for AI economics:
  • Quantify long-run economic value: test over longer horizons and differing market regimes, include transaction costs and market impact.
  • Robustness to adversarial manipulation: evaluate whether adversaries can craft evidence-text or beliefs to game the triangular score.
  • Cost-benefit modeling: compare additional compute and human oversight costs against reduced drawdowns, lower tail losses, and improved generalization.
  • Extend to other stochastic policy domains (monetary policy simulators, healthcare resource allocation) where process verification could reduce harmful specification gaming.

Caveats - Short test windows (Jul–Oct 2025) and training on A-share only may limit claims about long-term robustness. - Dependence on particular embedding and judge models (BGE-M3, Doubao-seed-1.8) and a large policy LLM (Qwen3-8B) may affect reproducibility and costs. - Risk of over-reliance on the judge model: if the judge has systematic biases, the semantic gate could introduce new failure modes.

Overall, Trade-R1 offers a principled, implementable approach to mitigate reward-hacking in RL for economic decision-making by integrating process-level verification into reward design, with clear implications for safer, more generalizable economic AI agents.