LLMs primed with tenants' transaction-cost perceptions more faithfully reproduce survey responses on energy-efficient renovations than demographic-only prompts, with gains observed across GPT-3.5 and two fine-tuned open models; persona-grounded prompting offers a theory-linked path to more interpretable policy simulation.

Main Finding

Grounding LLM-based tenant simulations in theory-driven perceived transaction-cost (PTC) personas (barriers + benefits) and an explicit PTC reasoning step substantially improves how well models reproduce real tenant survey responses about energy-efficient renovation (EER). This improvement holds across prompt-only and fine-tuned settings (GPT-3.5 prompt baseline; Mistral-8B and Llama-3.1-8B adapted via QLoRA), and the approach provides a interpretable bridge between transaction-cost policy theory and LLM-based policy simulation.

Key Points

• Problem addressed: Existing LLM-based persona simulations often condition on demographics only and miss practical/cognitive/social frictions (information burden, administrative effort, coordination, uncertainty) that shape policy responses.
• Conceptual contribution: Introduces PTC-aware persona modeling — representing tenants by empirically derived benefit and barrier personas (BE1–BE5, BA1–BA7) plus an explicit short reasoning checklist about burden, uncertainty, personal gains, and likely direction before producing an answer.
• Prompt design: Each instance combines a system instruction, an assigned PTC persona prompt, a PTC reasoning prompt, the verbatim survey question, and a required (2–4 sentences) + (single integer Likert) output format.
• Modeling strategies compared:
  • Prompt-only GPT-3.5-turbo under three prompt conditions (no PTC, PTC persona, persona + reasoning).
  • Open-weight local models (Mistral-8B-Instruct, Llama-3.1-8B-Instruct) adapted with QLoRA under:
    • Supervised fine-tuning (SFT) to reproduce survey reasoning traces + labels.
    • Group Relative Policy Optimization (GRPO) — a reward-guided policy update that normalizes rewards within sampled groups (no learned value model) and includes a KL penalty to the reference policy.
• Reward in GRPO: weighted sum of (i) ordinal accuracy (smoothly decreasing with distance from true label), (ii) output-format validity, and (iii) length control for the reasoning trace and answer.
• Evaluation metrics: MAE (ordinal error), exact-match accuracy, relaxed accuracy (within ±1).
• Empirical result (summary): Including PTC personas and the PTC reasoning prompt consistently lowers MAE and increases (exact and relaxed) accuracy compared to demographic-only prompting; fine-tuning and reward-guided adaptation further refine outputs. (Paper provides detailed experimental comparisons.)

Data & Methods

• Data:
  • Survey of tenants in the Netherlands: 1,068 respondent records, ≈40,548 question–answer pairs (≈24,242 attitude-related; rest demographic).
  • For fine-tuning: 4,600 attitude-related Q–A pairs (~20% of attitude subset) used for training; remainder held out for evaluation.
  • Each record includes socio-demographic context used to assign benefit and barrier personas via a persona predictor developed in prior work on Dutch social-housing tenants.
  • Persona distribution and full persona definitions available in appendix; code and data linked in paper repositories.
• Prompt & persona construction:
  • Each respondent assigned one benefit persona (BE1–BE5) and one barrier persona (BA1–BA7) inferred from contextual covariates rather than ad hoc descriptors.
  • PTC reasoning checklist nudges the model to consider burden, uncertainty, personal gains, and likely direction before answering.
• Models & training:
  • Prompt-only baseline: GPT-3.5-turbo under progressively richer prompts.
  • Open models: Ministral-8B-Instruct and Llama-3.1-8B-Instruct; adapters trained with QLoRA.
  • SFT objective: maximize likelihood of labeled survey responses (including reasoning trace).
  • GRPO: group-normalized advantage, KL penalty to reference policy; reward composed of accuracy, format, and length components.
• Evaluation:
  • Individual-response level: MAE, exact accuracy, Acc±1.
  • Comparisons across prompt-only vs. persona+reasoning and across SFT vs. GRPO adaptations.

Implications for AI Economics

• Theory-grounded persona conditioning matters: Incorporating institutional and behavioral theory (transaction-costs) into LLM persona design improves fidelity of agent-based or synthetic-survey simulations. For AI economics, this shows value in encoding mechanism-level frictions (search cost, administrative burden, coordination) rather than relying on demographics alone.
• Mechanism-aware policy simulation: PTC personas + explicit reasoning traces produce more interpretable agent responses, enabling richer counterfactual and subgroup analyses (e.g., how reducing administrative burden affects uptake across persona types).
• Practical modeling pipeline: Using QLoRA adapters with SFT and reward-guided GRPO provides a resource-efficient path to align open models to survey data and desired output formats — useful for economists who need calibrated synthetic agents but lack large compute budgets.
• Usefulness and limits for policy design:
  • Use cases: rapid scenario testing, mechanism-level sensitivity analysis, generating conditional predictions for policy design (e.g., targeting informational interventions vs. financial subsidies).
  • Cautions: synthetic respondent outputs are not a replacement for human survey data. Issues remain—representativeness (single-country, social-housing tenants), prompt/model sensitivity, potential biases from persona predictors, and ethical considerations when using simulated populations for policy decisions.
• Research directions: extend PTC-persona approach to broader populations and policy domains; calibrate and validate synthetic agents against external behavioral outcomes (not only survey labels); combine with causal inference frameworks to support policy counterfactuals; explore alternative reward designs and interpretability checks to ensure robust subgroup inferences.

If you want, I can extract and summarize the paper’s quantitative results (MAE / accuracy numbers and model-wise comparisons) from the results section and present a table of performance gains from adding PTC personas and from SFT vs GRPO.