Pairing a forecasting model with structured evidence and verification measurably improves AI-generated risk reports for South Korea's jeonse deposit guarantees, boosting detection of high-risk months while maintaining competitive average error; a 51-person practitioner review found the reports useful and recommended an operational pilot.

Main Finding

Converting LLM-assisted forecasts into auditable operational reports for housing-guarantee monitoring requires pairing a tail-sensitive forecaster with structured, pre-verified evidence. The authors build a pipeline that (1) uses an interpretable Temporal Fusion Transformer (TFT) trained with a regret-style loss to emphasize upper-tail detection, (2) retrieves historical precedents by matching the forecaster’s reasoning (variable-selection and temporal-attention) via centered kernel alignment (CKA), and (3) exposes a typed evidence contract that constrains LLM drafting and a deterministic audit that verifies claims before analyst review. This combination improves high-risk detection and yields LLM reports with higher numerical fidelity, claim grounding, and practical usefulness (endorsed by a 51-analyst practitioner evaluation).

Key Points

• Problem: Operational monitoring needs auditable, evidence-grounded warnings for sparse upper-tail events under data-governance limits; average-error-optimised models are insufficient.
• Pipeline overview:
  • Forecasting backbone: TFT that outputs forecast plus variable- & temporal-importance matrices.
  • Risk-sensitive loss: regret-style objective with heavier penalty for under-predicting high risk.
  • Retrieval: compute a model-reasoning matrix Z = [variable-selection ∥ temporal-attention] and retrieve analogous historical windows by CKA similarity in this reasoning space (not raw covariates).
  • Evidence contract: deterministic typed fields (forecast, baseline, direction, drivers, timing, analogues, high-risk references, caveats, sample support) fixed before generation.
  • Interfaces: raw (unstructured), card (evidence contract + raw), structured (card + prescribed report slots).
  • Generation + audit: LLMs draft using only admissible evidence; deterministic checks verify numbers, units, periods, labels, analogues, and sample-support before analyst review.
• Experimental findings:
  • Data: monthly South Korean jeonse deposit-guarantee panel (Sept 2015–Dec 2025).
  • Forecasting: TFT trained on original panel achieves better high-risk detection while maintaining competitive average error.
  • Reporting: experiments use synthetic aggregate scenarios calibrated to empirical ranges/temporal structure. Across eight LLMs, supplying structured evidence (card/structured interfaces) consistently improves report quality, numerical fidelity, and grounding versus raw context.
  • Practitioner study: 51 analysts and related professionals largely found the reports practically useful and supported piloting the system in operations.
• Core claim: Reliable LLM-assisted reporting is not just better LLM prompting — it requires (i) prediction models that expose reasoning, (ii) retrieval aligned with that reasoning, (iii) evidence contracts that bound claims, and (iv) deterministic audits plus analyst oversight.

Data & Methods

• Data
  • Monthly panel of region-by-housing-type segments in South Korea’s jeonse deposit-guarantee market.
  • Outcome: segment-level guarantee accident rate (accidents / completed guarantees) for t and one-month-ahead forecasts.
  • Time span: Sept 2015 — Dec 2025.
• Forecasting
  • Model family: Temporal Fusion Transformer (TFT), chosen for multi-horizon forecasting and internal variable-selection & temporal-attention outputs.
  • Loss: regret-style objective L_reg that weights over- and under-prediction asymmetrically (cost weights cover/under and λ scalar increase penalty for under-prediction) to prioritize recall of high-risk episodes.
  • Comparators mentioned: LSTM, DLinear, LightGBM.
• Model-reasoning representation & retrieval
  • Build Z_i,t = [M_VSN ∥ M_Attn] (lookback-length × (num_features + num_heads)) per window.
  • Candidate windows restricted to what would have been known at that monitoring date (no future leakage).
  • Similarity measure: centered linear CKA between Z matrices; highest CKA scores returned as analogous cases; high-risk references selected separately based on realized outcomes + exposure support.
• Evidence contract & interfaces
  • Typed fields E_i,t = {e_fcst, e_base, e_dir, e_drv, e_time, e_sim, e_risk, e_cav} computed deterministically pre-generation.
  • Interfaces:
    • I_raw = raw target + retrieved windows,
    • I_card = I_raw ⊕ E (evidence card),
    • I_str = I_card ⊕ S (structured report plan mapping evidence fields to report slots).
  • LLMs are restricted to verbalize only fields in E and not to introduce new forecasts, drivers, periods, or analogues.
• Generation & audit
  • LLM produces draft r given interface; deterministic audits then check label consistency, numerical values/units, period alignment, feature validity, analogues vs. references, and sample-support warnings.
  • Soft rubric-based checks (LLM-as-judge) may assess relevance/faithfulness, but final decision rests with human analysts.
• Evaluation
  • Forecast evaluation: average error + tail-sensitive monitoring utility (recall/operational regret for upper-tail).
  • Reporting evaluation: automatic fidelity checks and human evaluations across eight LLMs; practitioner study with 51 domain professionals.
  • Reporting experiments used synthetic aggregate scenarios calibrated to empirical ranges and temporal dynamics while forecasting backbone trained on original panel.

Implications for AI Economics

• Design for decisions, not only averages: Monitoring systems in public-finance contexts should optimize for tail-event visibility (asymmetric losses) because missed tail events have outsized operational costs.
• Leverage model-internal representations for retrieval: Matching precedent by the model’s reasoning (variable & temporal attention) via representation-based similarity (CKA) yields more relevant, admissible analogues than raw-covariate matching — useful wherever evidence must trace back to model logic.
• Evidence contracts reduce hallucination risk: Constraining LLMs to precomputed, typed evidence fields (and verifying them with deterministic audits) materially improves numerical fidelity and grounding; this pattern generalizes to other regulated domains where raw data cannot be exposed.
• LLMs as constrained drafters, not estimators: Treating LLMs as translators of verified evidence into readable reports preserves human accountability and avoids delegating estimation to a model that may hallucinate.
• Evaluation should include tail-sensitive metrics and human-in-loop assessments: AI-economics research and operational pilots should measure performance on decision-aligned objectives (e.g., recall of high-risk cases, operational regret) and include domain experts in evaluation.
• Governance & deployment: The workflow respects data-minimization constraints by not passing raw records to external models and provides a clear audit trail — a template for public institutions adopting AI-assisted monitoring.
• Limitations & open issues:
  • Reporting experiments relied on synthetic aggregate scenarios (calibrated to data) rather than full live deployment; generalization to other markets needs testing.
  • Model reasoning is interpretive, not causal — retrieved analogues are precedents, not causal proof.
  • Operational tuning (loss weights, CKA thresholds, exposure thresholds, and audit rule design) requires institution-specific calibration.
• Recommendation for practitioners/researchers: combine interpretable forecasting that exposes reasoning, representation-aligned retrieval (e.g., CKA on rationale matrices), typed evidence contracts, deterministic audits, and analyst-in-the-loop workflows when deploying LLM-assisted reporting in regulated, high-stakes economic monitoring tasks.