Augmenting tabular housing-placement models with fine-tuned LLM casenote summaries raises accuracy and narrows error disparities in a nonprofit dataset; zero-shot LLM use produces mixed fairness effects and findings are limited by short, redacted notes and single-site data.

Main Finding

Fine-tuning large LLMs (Llama 3 70B) on serialized tabular records for housing-placement prediction substantially improves overall accuracy (up to 70.2%) but can amplify subgroup disparities (worse Equality-of-Opportunity gaps). Augmenting tabular prompts with LLM-generated casenote summaries improves zero-shot performance (notably for smaller models) and—importantly—can reduce fairness gaps when used carefully. Explicitly emphasizing historically predictive tabular features (e.g., max prior placement) increases accuracy but risks reinforcing historical bias; effects vary by model size and fine-tuning.

Key Points

• Dataset and task

  • 471 clients from a street-outreach nonprofit (seen monthly 2019–2021).
  • Outcome Y ∈ {0,1,2,3}: highest housing placement by 2021 (0 = streets, 3 = permanent supportive housing).
  • Tabular covariates include demographics (gender, race, age range), outreach metadata, binary casenote-extracted indicators, count of prior engagements, and max prior placement (identified as most predictive).
  • Casenotes are short and heavily redacted.

• Models evaluated

  • Llama 3 variants: 17B and 70B (Together AI API), tested zero-shot and fine-tuned.
  • Random forest baseline on tabular covariates.
  • Prompt format: tabular record serialized to natural language; optional LLM-generated casenote summary; optional prompt cue for top feature.

• Fairness metrics

  • Multiclass Statistical Parity (SP) and Equality of Opportunity (EoO), reported as mean and max classwise deviations across race (White, Black, Other) and gender (Male, Female).
  • Evaluations restricted to subgroups with sufficient sample size.

• Quantitative highlights (selected)

  • Fine-tuned 70B (no summaries): accuracy 70.2%, EoO mean ≈ 0.071, EoO max ≈ 0.209 (accuracy up, fairness gaps larger).
  • Fine-tuned 70B + summaries: accuracy 68.6% (slight drop), EoO mean ≈ 0.021, EoO max ≈ 0.097 (notable reduction in EoO disparities).
  • Zero-shot 17B: accuracy 33.1% → with summaries 47.0% (≈+13.9 pp), EoO mean reduced for some variants.
  • Random forest baseline: accuracy 61.2% but worse SP metrics and comparable EoO to some LLMs.

• Feature-importance cueing

  • Adding the top tabular feature (max placement before 2019) to prompts raised accuracy (especially zero-shot 17B), but often increased EoO disparities—suggesting that explicitly surfacing historically predictive features can propagate structural bias.
  • Interactions depend on model capacity: 17B and 70B responded differently to feature cues and summaries; larger/fine-tuned models can have learned balances that are disrupted by overemphasis on a single feature.

• Other observations

  • Zero-shot LLMs tended to predict class 2 more often (a "safe" middle-class bias).
  • Female TPRs were consistently lower than male TPRs across models (dataset male:female ≈ 4:1).
  • The authors explored alternative splits, SMOTE oversampling, prompt optimization, and few-shot prompting; none consistently improved both accuracy and fairness.

Data & Methods

• Data: 471 client records, casenotes (short, redacted), tabular baseline covariates; outcome is highest placement level by 2021.
• Serialization: Tabular fields (and optionally casenote summaries and feature cues) converted to a natural-language prompt for LLM input.
• Models:
  • Llama 3 17B and 70B, evaluated zero-shot and via fine-tuning (train/val/test primarily 75/10/15).
  • Random forest trained on tabular covariates as baseline.
• Casenote summaries: generated by LLM and appended to the prompt in augmented variants.
• Fairness evaluation:
  • Multiclass Statistical Parity: per-class deviation of predicted rates from overall prediction rates (reported as SP mean and SP max).
  • Multiclass Equality of Opportunity: per-class TPR deviations from marginal TPR (reported as EoO mean and EoO max).
  • Protected attributes: race (White/Black/Other) and gender (Male/Female).
• Additional experiments: prompt engineering, feature cueing (top feature and up to 10 additional features), alternative train/test splits, SMOTE oversampling; authors report mixed/unstable gains.

Implications for AI Economics

• Performance–fairness tradeoffs matter in resource-allocation contexts. Higher aggregate accuracy from fine-tuning can come at the cost of increased disparities; cost-benefit analyses of deploying tuned LLMs must include subgroup welfare impacts (who benefits/harmed).
• Value of unstructured information: LLM summarization can extract useful contextual signals from casenotes that improve zero-shot performance at low implementation cost and—crucially—can mitigate some fairness gaps. This suggests economically attractive investments in lightweight LLM-based feature extraction when structured data are limited.
• Model capacity and calibration: larger and fine-tuned models can better leverage nuanced context but are also more likely to amplify historical imbalances if training data are imbalanced. Deployment decisions should consider model capacity, fine-tuning regimen, and the distributional makeup of populations served.
• Caution on feature highlighting: surfacing historically predictive features (even non-protected ones) can improve predictive accuracy but may entrench inequities because those features correlate with protected group membership. Economists and policymakers should avoid naive reliance on historically predictive signals without considering fairness externalities.
• Practical recommendations for practitioners and policymakers:
  • Always audit multiclass fairness (SP and EoO) on relevant subgroups before deployment.
  • Prefer augmenting tabular models with LLM-derived summaries rather than only fine-tuning large LLMs, when the goal includes reducing disparities.
  • Monitor subgroup-specific TPRs and worst-case gaps (EoO max), not just overall accuracy.
  • Where fine-tuning is used, mitigate imbalances (sampling, fairness-aware objectives, separate calibrations) because fine-tuning can disproportionately benefit majority groups.
  • Pilot with smaller models as robustness checks: smaller models may be less sensitive to noisy text inputs and can reveal failure modes.
• Research & policy directions:
  • Broader studies on longer, less-redacted casenotes and larger populations to test generalizability.
  • Develop fairness-aware LLM fine-tuning and prompt interventions that balance accuracy with subgroup parity in multiclass outcomes.
  • Economic evaluation of downstream impacts (allocation efficiency vs distributional harms) to guide acceptable tradeoffs in social-service deployments.

If you want, I can: (a) produce a one-page slide-ready summary; (b) extract the exact metric table rows into a compact CSV-style table; or (c) draft recommended operational auditing steps for a nonprofit deploying such a model. Which would be most useful?