0 cumulative citations
View corpus contextSplitting a financial-analysis prompt into lens-aligned specialists boosts LLM accuracy on mechanical numerical tasks but can hurt integrative judgments; targeted RL fine-tuning of a smaller model restores and improves judgement accuracy and transfers across firms and regulatory regimes.
From Numbers to Judgment: Specialist LLM Agents and Reinforcement Learning for European Listed Real Estate
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
Decomposing prompts into lens-aligned specialist agents substantially improves LLM performance on mechanical numerical extraction tasks (~+15.8 percentage points), while targeted RL post-training of a smaller model (GRPO on Qwen3.5-9B) boosts integrative judgment performance and generalizes to unseen firms and regulatory wrappers.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
We study whether the localized numerical operations and integrative judgments of financial analysis benefit from the same form of LLM specialization. Larix maps a 16-lens European listed-real-estate analysis framework to eight lens-aligned specialists; we compare a frontier LLM under monolithic versus specialist-decomposed prompting while holding the model, source evidence, task instructions, output schema, and scoring fixed. Across 19 firms spanning seven regulatory wrappers, decomposition improves the numerical-task aggregate by 15.8 percentage points but does not reliably improve, and can reduce, performance on judgment tasks, a pattern stable across four frozen-template dispatches; a single-agent control given the complete framework does not reproduce the numerical gain. Post-training Qwen3.5-9B with GRPO using task-aligned structured rewards then raises the development-split score by 12.0 points and the judgment aggregate by 14.2 points, with gains on all four sub-ceiling tasks; the gains transfer to unseen firms (+15.2 points overall; +40.4 on covenant stress) and to unseen regulatory wrappers (+4.3), with positive transfer on all three anti-memorization splits. Prompt-level decomposition thus improves modular numerical execution, whereas targeted parameter adaptation improves integrative financial judgment.
Summary
Main Finding
Specialist (lens-aligned) prompt decomposition materially improves localized numerical tasks in regime-aware financial analysis, but it does not reliably improve—and can reduce—integrative judgment performance. By contrast, targeted parameter adaptation (GRPO post-training of a smaller model) substantially improves integrative financial judgment and transfers to unseen firms, wrappers, and periods. In short: prompt-level modularization helps numerical execution; RL fine-tuning (task-aligned structured rewards) helps cross-evidence integrative judgment.
Key Points
- Experiment design isolating decomposition: same frontier LLM, identical source evidence, task instructions, JSON output schema, and scorer; only the presentation (monolith vs full-framework vs lens-aligned specialist) varies.
- Tasks split into numerical (Tnum = {T1, T3, T6}) and judgment (Tjudg = {T2, T5}) classes:
- Numerical: closed, mechanically verifiable extractions/calculations (e.g., regime-appropriate metric, implied cap rate, payout-regime classification).
- Judgment: require reconciling multiple disclosures or regime rules (e.g., reconciliation adjustment, covenant assessment).
- Specialist decomposition (Larix: 16 lenses → 8 specialists; benchmark exercises 3 specialists) results:
- Numerical-task aggregate improved by +15.8 percentage points (19-firm continental cohort).
- Judgment-task aggregate unchanged or worse in primary dispatches; decomposition can hurt integrative tasks.
- A monolithic prompt that discloses the full 16-lens framework to one agent (Claude-Full) did not reproduce the numerical gains—scoped lens ownership, not mere disclosure, appears important.
- RL post-training (Qwen3.5-9B + GRPO with structured, task-aligned rewards):
- Development-split overall score increased by +12.0 points; judgment aggregate increased by +14.2 points.
- Gains observed on all judged sub-tasks and transferred out-of-distribution:
- Unseen firms: +15.2 points overall; covenant-stress judgment +40.4 points.
- Unseen regulatory wrappers: +4.3 points.
- Positive transfer on anti-memorization (later-period) splits.
- Practical architectural implication: prompt-level agent decomposition helps precise, localized computations; parameter-level adaptation (RL fine-tuning) is required to improve integrative, cross-evidence judgment.
Data & Methods
- Evaluation universe:
- 25 European listed-real-estate firms spanning 8 legal/reporting wrappers; main same-model comparison used a 19-firm continental cohort (95 task–firm instances across a 5-task benchmark).
- Primary-source evidence injected for judgment tasks and payout extraction; some numerical tasks were closed-book by design.
- Task set (evaluated tasks):
- T1: regime-specific operating metric (numerical, ±5% tolerance).
- T2: reconciliation adjustment (judgment, exact matching after alias normalization).
- T3: implied cap-rate components (numerical, ±15% tolerance).
- T5: covenant and breach assessment (judgment, joint AND semantics across fields).
- T6: payout-regime classification (numerical/extraction; categorical + payout ratio ±10%).
- Controlled comparisons:
- Frontier LLM (Claude Opus 4.8) evaluated under three prompting conditions:
- Claude-Mono: single general prompt.
- Claude-Full: monolith prompt with complete 16-lens framework disclosed.
- Claude-Spec: deterministic routing to the corresponding lens-aligned specialist prompt (no inter-specialist messaging during evaluation).
- Smaller model (Qwen3.5-9B) evaluated zero-shot (specialist prompts) vs post-trained with GRPO (LoRA adapter rank 32).
- Frontier LLM (Claude Opus 4.8) evaluated under three prompting conditions:
- Scoring & reward:
- Deterministic JSON schema, pre-declared field scorers and weights, numeric tolerances, exact/categorical alias normalization; invalid/unparsable outputs score zero.
- RL reward = exact deterministic task score (i.e., post-training optimizes the same rubric used for evaluation).
- RL training specifics:
- Training corpus: 195 benchmark-derived rows + 5 training-only counterfactuals = 200 rows; after held-out splits, 70 rows used for training (65 real + 5 counterfactual).
- LoRA adapter trained with GRPO (veRL + vLLM rollouts) on a single H100 GPU; per step: 64 sampled prompts, G = 8 candidates, temperature 1.0. Planned 90 steps but stopped early; evaluated step-20 checkpoint.
- Statistical protocol:
- K = 4 frozen-template dispatches (averaged per instance).
- Paired comparisons at firm level; 95% CIs from paired McNemar and sign-flip tests; Holm correction for per-task comparisons.
Implications for AI Economics
- Division of labor and returns to specialization:
- There are distinct, task-dependent returns to different kinds of specialization. Prompt-level modularization yields large gains for localized, procedural numeric tasks (high marginal return from narrowing context and explicit intermediate fields). Parameter-level adaptation (RL fine-tuning with structured rewards) yields gains for integrative judgments that require reconciling heterogeneous evidence.
- Economic implication: designing AI systems for complex economic domains is not a one-size-fits-all allocation of compute or model scale. Architectural choice (agentization vs. fine-tuning) matters and should align with the task mix.
- Cost-effectiveness and model-scale trade-offs:
- A relatively small 9B model post-trained with structured RL can outperform an unfine-tuned frontier model on integrative judgment tasks. This suggests that targeted fine-tuning (task-aligned rewards, LoRA-style adapters) is a cost-effective route to improve decision-relevant performance, rather than always scaling to larger frozen models.
- Evaluation and incentive alignment:
- Using deterministically structured rewards tied to the same evaluation rubric ensures alignment between training objectives and evaluation. For economically consequential domains (finance, regulation), this reduces risk of misaligned fluency (plausible but incorrect outputs).
- Policy/regulatory implication: auditors and deployers should require source-grounded, provenance-scored outputs and task-aligned validation when LLMs support regulatory or fiduciary decisions—especially where cross-jurisdictional regime rules are decisive.
- System design recommendations:
- Hybrid approach: employ specialist prompting (or extraction-focused modules) for high-assurance numeric computations and structured RL-adapted policies for integrative, cross-document judgments; but remain cautious—strict decomposition can withhold cross-lens evidence needed for integration.
- Design of multi-agent financial systems should consider where to (a) isolate computations, (b) allow cross-agent evidence flow, and (c) invest in parameter-level adaptation for global judgments.
- Limitations and future questions relevant to AI economics:
- The evaluation isolates the specialist layer and does not test downstream synthesis, conviction calibration, or market-impact decisions (position sizing). Future research should quantify how specialist outputs are combined and whether RL fine-tuning of synthesizer layers yields further gains.
- The RL corpus was small and training stopped early; larger, more diverse structured-reward corpora may yield stronger, more robust transfer. Understanding scaling laws for parameter-level specialization vs. prompt engineering remains an open question with economic implications for development and deployment costs.
- Risk of omission: narrow prompt scopes can omit cross-lens evidence. Economic deployments must monitor for edge cases where decomposition increases systemic error (e.g., misapplied regulatory rule across jurisdictions).
Overall, the paper provides controlled, actionable evidence that (1) prompt-level specialist decomposition is an effective, low-commitment lever to raise numerical-extraction reliability, and (2) structured RL fine-tuning is an effective means to improve integrative financial judgments and achieve out-of-distribution generalization — informing design and investment choices for AI systems in regulated economic domains.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper uses a carefully controlled experimental design (same model/evidence/instructions/scoring) and deterministic rubriced scoring with expert-blinded ground truth for many items, which supports causal attribution of prompt-level effects. However, the empirical scope is limited (five benchmark tasks, a 19-firm primary cohort, and a small RL training corpus with a truncated training run), some ground-truth labels were curated by the authors rather than exclusively external experts, and the downstream decision pipeline and real-world economic outcomes are not evaluated, constraining external validity.
Methods Rigormedium — Strong internal controls: identical evidence and scoring across conditions, frozen templates, deterministic JSON output schemas, paired statistical tests, and an expert-blinded subset of ground truth; explicit separation of prompt-level and parameter-level interventions. Limitations include small sample size, limited task breadth (3 numerical, 2 judgment tasks), partial reliance on author-curated labels, a single frontier model family and one small model family, and an interrupted/shortened RL run (only up to a step-20 checkpoint). The evaluation also omits end-to-end synthesis and real decision consequences.
SamplePrimary same-model comparison: 19 continental European listed-real-estate firms spanning six/seven regulatory wrappers (French SIICs, Austrian Immobilien-AG, Belgian GVV/SIRs, German AGs, Italian SIIQ, Spanish SOCIMIs, Dutch FBIs) producing 95 task–firm instances across five tasks; additional six Swiss AGs used for cross-wrapper analysis (full universe 25 firms). Tasks: three numerical (T1, T3, T6) and two judgment (T2, T5). Evidence regimes include closed-book probes for some numerical tasks and injected primary-source extracts for judgment/extraction tasks. RL corpus: ~200 training rows (195 benchmark-derived + 5 counterfactuals), with held-out splits for unseen firms (35 rows), unseen wrappers (25 rows), held-out periods (45 rows), and a 25-row development split; GRPO post-training performed on Qwen3.5-9B LoRA adapter (rank 32) with G=8 candidates, sampled from 70 training rows due to limited steps.
Themesproductivity human_ai_collab
IdentificationA within-model controlled comparison that holds the LLM, source evidence, task instructions, output schema, scoring, decoding settings, and frozen dispatch templates constant while varying only (a) the prompt presentation (monolithic vs. full-framework disclosure vs. lens-aligned specialist prompts) and (b) model parameters (zero-shot Qwen3.5-9B vs GRPO post-trained checkpoint). Effects are estimated as paired differences at the firm level across the same task instances, with permutation/sign-flip tests and paired McNemar statistics to assess significance; RL effects are isolated by training on a disjoint training set and evaluating on held-out firms, wrappers, and periods.
GeneralizabilityDomain-specific: results apply to European listed real estate and the particular 16-lens Larix framework; may not generalize to other financial sectors or non-financial domains., Limited task scope: benchmark covers five tasks (three numerical, two judgement); effect sizes may differ for broader or more complex tasks., Model families: frontier results reported on a single high-end model (Claude Opus 4.8) and one small model family (Qwen3.5-9B); findings may not hold for other architectures/sizes., Training scale: RL post-training used a small training corpus and a truncated number of update steps (checkpoint at step 20), limiting conclusions about parameter-level adaptation at scale., No end-to-end evaluation: downstream synthesis, conviction calibration, and economic decision outcomes (e.g., trading performance, analyst productivity) were not measured., Partial label provenance: some ground truth rows were authored by the paper's authors rather than exclusively by independent experts, introducing possible bias.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Lens-aligned specialist decomposition improved the numerical-task aggregate by 15.8 percentage points relative to monolithic prompting in the frontier-model comparison. Output Quality | positive | Aggregate performance on regime-specific metric extraction, implied cap-rate calculation, and payout-regime classification tasks |
Reading fidelity
high
Study strength
medium
|
n=19
15.8 percentage points
|
| Specialist decomposition did not improve the judgment-task aggregate in the primary dispatch, and the paper reports that it can reduce performance on judgment tasks. Decision Quality | null_result | Aggregate performance on reconciliation-adjustment and covenant/breach-assessment judgment tasks |
Reading fidelity
high
Study strength
medium
|
n=19
|
| Providing the complete 16-lens framework to a single monolithic model invocation did not reproduce the numerical performance gain achieved by scoped specialist prompting. Output Quality | negative | Numerical extraction-task performance, particularly the extraction tasks |
Reading fidelity
high
Study strength
medium
|
n=19
|
| GRPO post-training of Qwen3.5-9B with task-aligned structured rewards increased the development-split score by 12.0 points and the judgment-task aggregate by 14.2 points. Decision Quality | positive | Overall five-task development-split score and judgment-task aggregate |
Reading fidelity
high
Study strength
medium
|
n=25
12.0 points overall; 14.2 points on the judgment aggregate
|
| The post-training gains occurred on all four sub-ceiling tasks evaluated in the development split. Decision Quality | positive | Per-task structured benchmark scores after post-training |
Reading fidelity
high
Study strength
medium
|
n=25
|
| Post-training gains transferred to unseen firms, producing a 15.2-point overall gain and a 40.4-point gain on the covenant-stress task. Decision Quality | positive | Overall and covenant-stress judgment performance on firms unseen during training |
Reading fidelity
high
Study strength
medium
|
n=35
15.2 points overall; 40.4 points on covenant stress
|
| Post-training gains transferred to unseen regulatory wrappers, with a 4.3-point improvement. Decision Quality | positive | Structured financial-analysis performance on regulatory wrappers unseen during training |
Reading fidelity
high
Study strength
medium
|
n=25
4.3 points
|
| The post-training gains were positive on all three anti-memorization evaluation splits: held-out firms, held-out regulatory wrappers, and held-out reporting periods. Decision Quality | positive | Generalization performance across unseen firms, regulatory wrappers, and later reporting periods |
Reading fidelity
high
Study strength
medium
|
n=105
|
| The primary same-model comparison used a 19-firm continental cohort and generated 95 task-firm instances per prompting condition. Other | other | Benchmark evaluation coverage |
Reading fidelity
high
Study strength
high
|
n=95
95 task-firm instances per condition
|
| The evaluation framework did not empirically test downstream cross-agent synthesis, conviction calibration, or position sizing. Task Allocation | null_result | Downstream synthesis, conviction calibration, and position-sizing performance |
Reading fidelity
high
Study strength
high
|
not reported
|