An open-weight LLM fine-tuned with GRPO and a safety-gated finance rubric is estimated—via a judge-independent doubly-robust off-policy audit—to double the gross-profit lift of leading commercial models while reducing downside and tail risk, though the result is based on observational logged data rather than randomized deployment.

Main Finding

Fine-tuning an open-weight LLM (Qwen3.5-27B) with Group Relative Policy Optimization (GRPO) using a finance-grounded, safety-gated LLM-as-judge rubric produces recommendations that (a) score best on the rubric and (b) — by a judge-independent doubly-robust CATE audit — yield roughly double the estimated gross-profit lift of the best commercial baseline (0.0228 vs. 0.0104), with lower downside rate and less negative tail risk. The paper argues that a causal, judge-independent audit is a necessary complement to LLM-as-a-judge evaluation to detect reward-hacking and to measure true business value.

Key Points

• Task framing: financial advice generation (single JSON recommendation with action, reasoning, expected_impact) formulated as an RL problem optimizing a scalar reward rather than supervised imitation of historical decisions.
• Training algorithm: Group Relative Policy Optimization (GRPO) over groups of K=12 samples, with KL anchoring to the base model (β = 0.001) and LoRA fine-tuning of Qwen3.5-27B (27B) under DeepSpeed ZeRO-2.
• Reward / judge: Claude Opus 4.5 used as a rubric judge that scores 11 binary criteria (1 safety gate + 10 quality criteria across specificity, actionability, data grounding, reasoning, impact, relevance). Final reward R(a) = c(a) − p(a) where c(a) is safety-gated average of the 10 quality bits (or zero if unsafe) and p(a) is a thinking-length penalty (0–0.2); unparsable JSON receives −0.5.
• Judge evaluation (rubric score, mean of 10 quality criteria scaled to [0,10], n=2,500 per model): Qwen3.5-27B-GRPO = 9.514 [9.505,9.524]; Claude Opus 4.6 = 9.365; GPT-5.4 = 8.949; untrained Qwen3.5-27B base = 8.457.
• Judge-independent causal audit: map free-text recommendations to a discrete action catalogue (60 actions; 49 passed eligibility), then estimate per-action Conditional Average Treatment Effects (CATEs) on year-over-year gross-profit growth with a doubly-robust AIPW estimator. Propensity = multi-label MLP over a 768-dim state embedding; outcome model = neural net with treated/control heads. Held-out scoring uses stratum-level AIPW estimates by propensity decile.
• Causal-audit evaluation protocol: 500 held-out business states × 5 independent runs per policy (2,500 samples per policy); reported metrics are mean DR-CATE (estimated lift), downside rate (fraction of recommendations mapping to negative-effect strata), and CVaR0.10 (mean worst 10%).
• Causal audit results (means and interpretation):
  • Qwen3.5-27B-GRPO: lift = 0.0228 [0.0211,0.0246], downside = 0.155, CVaR0.10 = −0.073 — best on all three metrics.
  • Strongest commercial: Claude Opus 4.6: lift = 0.0104, downside ≈ 0.232, CVaR0.10 ≈ −0.100.
  • Other baselines: GPT-5.4 lift ≈ 0.0082 (noisiest); Claude Sonnet 4.5 ≈ 0.0028; Claude Opus 4.5 estimated negative (≈ −0.0025).
• Effect of GRPO fine-tuning: starting base model lift = 0.0170 → GRPO lift = 0.0228 (≈ +34% relative), statistically significant (p-values reported). Downside and tail risk also improved.
• Important divergence: the judge and causal audit do not rank baselines identically (e.g., untrained base ranks last per judge but second by causal audit), supporting the audit as an independent signal rather than a duplicate of rubric scoring.

Data & Methods

• Data: logged business financial states (company-month-level financial summaries), with PII replaced by realistic synthetic surrogates. Target KPI: year-over-year gross-profit growth (used as outcome in causal audit). State encoded as 768-dim sentence embedding.
• Model & training: base = Qwen3.5-27B, LoRA fine-tuned with GRPO (K=12 candidates per prompt, max completion 8,000 tokens, KL β = 0.001, learning rate 5e−5 cosine, doubly-robust GRPO loss variant). Selection by validation reward.
• Rubric: 11 binary criteria; safety gate hard-zeros reward if recommendation would cause significant harm. JSON outputs required; parse-failure penalty −0.5.
• Causal audit pipeline:
  • Action mapper: independent LLM classifier maps free-text recommendations to a fixed action catalogue; low-confidence/no-match recommendations excluded from scoring.
  • Propensity model: multi-label MLP predicting P(A=1 | x) per action.
  • Outcome model: neural net with treated/control heads predicting Y (YoY gross-profit growth).
  • Estimator: doubly-robust AIPW pseudo-outcome computed on logged datasets; per-action × propensity-decile strata aggregated and used as lookup for held-out recommendations.
  • Metrics computed over 500 held-out states × 5 runs per policy; mapper, propensity, and outcome models fixed across policies (ensuring comparable scoring).
• Reproducibility notes: authors will release a synthetic, anonymized mapper and evaluation artifacts.

Implications for AI Economics

• Demonstrates that task-specific RL fine-tuning of a mid-sized open model (with careful reward design and safety gating) can outperform larger commercial LLMs on a narrow, high-value business objective (estimated gross-profit lift). This has implications for the cost-benefit tradeoff between: (a) paying for commercial LLM APIs and (b) investing in in-house fine-tuning of open models tailored to firm objectives.
• Importance of economically meaningful evaluation: LLM-as-a-judge rewards polish and alignment with stylistic rubric criteria but can be gamed or misaligned with business outcomes. A causal, judge-independent audit (even observational) is a valuable complement to detect such reward-hacking and to quantify estimated business value.
• Methodological template for firms: (1) design domain-grounded rubric with safety gate; (2) fine-tune via GRPO or RL-from-AI-feedback; (3) audit via action mapping + doubly-robust off-policy estimators to estimate KPI lift and downside risk before deployment. This pipeline reduces reliance on expensive human labels and provides an actionable risk/benefit estimate.
• Cautions and economic limits:
  • Observational audit assumptions: AIPW is unbiased only if either propensity or outcome model is correct and if actionable confounding is sufficiently captured by the state embedding. Unobserved confounding / selection bias can distort estimated CATEs; estimates are not substitutes for randomized trials (A/B tests).
  • Action mapping and exclusion: mapping errors and no-match exclusions can bias which recommendations are audited; the audit reflects only mapped, supported actions.
  • Judge-in-the-loop bias: training used the same family of judge LLMs that also produced rubric scores, creating an expected advantage in judge metrics; causal audit helps mitigate this but cannot fully replace prospective randomized evaluation.
  • External validity: results are specific to the dataset, action taxonomy, and gross-profit KPI; different business populations, KPIs, or longer-horizon outcomes might change conclusions.
• Practical next steps for decision-makers:
  • Run randomized A/B tests of the GRPO policy in controlled pilots to validate observational audit findings.
  • Cost analysis: compare API costs for commercial LLMs vs. engineering and inference costs of a tuned open model (LoRA + GRPO) at production scale.
  • Expand audit metrics: incorporate additional KPIs (cash flow, churn, long-term survival) and longer horizons to capture potential delayed harms or benefits.
  • Invest in robust action mapping, richer covariates to reduce confounding, and transparent safety gates to limit downside exposure and regulatory risk.

Limitations highlighted by the paper (summary): reliance on an LLM judge for training, potential reward-hacking, observational nature of the causal audit (not a randomized deployment), mapping exclusions, and dataset-specificity. The approach nonetheless provides a practical, economically oriented framework for producing and assessing financial advice from LLMs.