How you present model evidence matters more than which LLM you use: in mortgage credit scoring, the evidence modality (tabular, network or both) explains far more variation in explanation fidelity than the choice of LLM; language models reliably name influential factors but frequently err on the direction of influence, and credit professionals judge outputs more harshly than lay readers.

Main Finding

The paper shows that LLMs can turn technical post-hoc explanation artefacts (SHAP, GNNExplainer) into fluent, stakeholder-oriented risk narratives, but the dominant determinant of explanation quality is the representation of evidence (modality/pipeline) rather than the choice of LLM. Narratives reliably surface influential factors but frequently fail on the direction of influence; professional evaluators apply stricter evidentiary standards than non-professionals. These results reframe the design problem from selecting LLMs to designing evidence representations and governance around directional fidelity.

Key Points

• Three explanation pipelines were evaluated: tabular (XGBoost + SHAP), network (GAT/GNN + GNNExplainer), and bimodal (both).
• LLM configurations compared: Gemma 3 (4B, fine-tuned), DeepSeek R1 (70B, fine-tuned), and Gemini 2.5 (commercial, zero-shot).
• Evaluation combined automated evidence-grounding checks and a human study comparing credit-risk professionals and non-professionals across multiple decision-relevant dimensions (e.g., fidelity, trustworthiness, communicability, usability).
• Main empirical findings:
  • Evidence modality (pipeline) explains far more variance in evidence-grounding scores than the LLM choice — the representation of evidence is the binding constraint on explanation quality.
  • Generated narratives typically name the influential factors correctly but are less reliable about the direction (positive vs negative influence), a salient failure mode for adverse-action communications.
  • Professionals judge explanations by stricter evidentiary standards than non-professionals; rater heterogeneity often exceeds system differences.
• Practical contributions: a separable three-stage architecture (prediction → explainer → LLM narration); concrete prompt and post-processing design to constrain hallucination/overclaim; empirical decomposition of sources of explanation quality.

Data & Methods

• Data: Freddie Mac single-family loan-level data (origination cohorts 2015 for training, 2016 for held-out evaluation); loans retained with ≥6 months servicing history; standard preprocessing (imputation, recoding, scaling).
• Network construction: static origination-stage graphs (6-month and 1-month windows). Nodes = loans; edges connect loans sharing zip-code prefix or lender (edge types one-hot encoded); edges form bidirectional cliques per grouping.
• Predictive models:
  • Tabular: XGBoost classifier (hyperparams via grid search, 5-fold CV, ROC-AUC selection).
  • Network: Graph Attention Network (GAT) with edge-aware transformations and neighbour sampling; class imbalance handled by ~1:5 negative subsampling.
• Post-hoc explanation methods:
  • SHAP for XGBoost (feature-level attributions).
  • GNNExplainer for GAT (local explanatory subgraph + node-feature importance).
• LLM narration layer:
  • Evidence blocks: rule-based post-processing of explainer outputs (rank top-k, percentiles, plain-language descriptors, preserve directional info).
  • Prompt design: shared system prompt enforcing objectivity + pipeline-specific user prompts (stepwise reasoning, explicit subgoals, constrained output formats, counterfactual-style components).
  • LLMs: two open models fine-tuned on domain data (Gemma 3 4B, DeepSeek R1 70B) and one commercial zero-shot model (Gemini 2.5).
• Evaluation:
  • Automated checks for evidence-grounding, coverage, and directional alignment.
  • Human study contrasting professionals vs non-professionals on eight decision-relevant dimensions (including fidelity, trustworthiness, communicability, usability).
  • Variance decomposition attributing differences to evidence modality, LLM, and evaluator cohort.

Implications for AI Economics

• Evidence representation matters more than LLM choice for deployed explanation quality. Economic analyses of model deployment should therefore allocate engineering and governance resources to how model evidence is represented and constrained, not only to LLM procurement.
• Directional errors are governance-relevant and can produce costly adverse-action mistakes. From a regulatory-cost perspective, firms face greater compliance and litigation risk if explanations misstate directionality even when naming correct factors. Economic models of regulatory risk should value directional fidelity highly.
• Bimodal/alternative-data (network) approaches can improve predictive performance but complicate explainability. Cost–benefit assessments of adopting relational/alternative data must include the incremental expense of building robust, audit-ready evidence representations and human-evaluation pipelines.
• LLMs can reduce the operational cost of generating stakeholder-appropriate narratives (lowering time/effort for documentation, customer communications, and validation), but that saving is conditional on rigorous constraints, provenance tracking, and domain fine-tuning. Economic gains from LLMs should be discounted by governance, monitoring, and validation costs.
• Heterogeneous stakeholder standards imply that evaluation metrics for explanation systems must be cohort-aware. When estimating the value of explainability investments, economists should model multiple utility functions (e.g., internal risk teams prioritize fidelity; customers prioritize clarity).
• Market and policy implications: regulators and standard-setters should emphasize evaluation standards focused on evidence grounding and directional correctness. This affects how firms invest in compliance infrastructure and how the economics of model risk management are internalized.
• Research and product prioritization: investments that improve evidence extraction from non-tabular sources (stable, provenance-rich explainers) likely yield larger returns in explanation quality than marginally larger or more expensive LLMs.

Suggestions (operational): - Prioritize robust evidence-block design, constrained prompting, provenance/versioning, and domain fine-tuning for any LLM-based explanation layer in credit settings. - Use professional evaluators in validation loops and treat directional correctness as a primary failure mode to mitigate regulatory risk. - When evaluating alternative-data models (networks/GNNs), include explainability overhead in cost models—both technical (explainers, post-processing) and governance (human review, documentation).