Large language models match experts when ranking policy criteria but disagree on scoring and final rankings in Ghana’s renewable-energy MCDA, implying LLMs are valuable for elicitation but need expert oversight and hybrid workflows to produce reliable policy recommendations.

Main Finding

LLMs can effectively approximate domain experts in selected stages of an expert-driven SWOT-based MCDA for renewable energy policy in Ghana—especially in criteria weighting—yet they produce more divergent outputs in decision matrices and final policy rankings. Consequently, LLMs are useful as decision-support tools for elicitation and preliminary scoring, but full MCDA pipelines using LLM outputs require expert validation and hybrid workflows to ensure reliable policy recommendations.

Key Points

• Study context: renewable energy policy evaluation in Ghana using a SWOT-based Multi-Criteria Decision Analysis (MCDA) framework.
• Models compared: GPT 5.3‑mini, Flash 3.0, Grok 4.2, Sonar, Sonnet 4.6.
• Two MCDA stages where LLMs were evaluated:
• Criteria weighting via Ranking Comparison (RANCOM).
• Policy ranking via MARCOS (Measurement of Alternatives and Ranking according to Compromise Solution), aggregated using Compromise Fuzzy Ranking (CFR) for group consensus.
• Stability and robustness checks: repeated LLM evaluations and incorporation of Risk‑Informed Decision Making (RIDM).
• Empirical findings:
  • High agreement between LLMs and experts on criteria weights (RANCOM stage).
  • Greater divergence between LLMs and experts in decision matrices and resulting policy rankings (MARCOS stage); variation across LLMs.
  • Domain experts: top criterion = increasing electricity demand; top policy = national consensus on renewable energy implementation.
  • LLMs partially reproduce expert priorities but differ in ranking structures depending on model and repeated runs.
• Methodological contribution: systematic, within‑pipeline comparison of expert vs. LLM judgments, plus use of CFR to form group consensus and RIDM to assess stability.

Data & Methods

• Decision context: renewable energy policy alternatives for Ghana, evaluated using an expert-driven SWOT-based MCDA.
• Expert input: domain experts provided criteria, weights, and policy assessments used as a baseline.
• LLM procedure:
  • Five LLMs prompted to perform identical elicitation and scoring tasks matching the expert MCDA protocol.
  • Repeated prompts used to measure intra-model variability.
• MCDA mechanics:
  • Criteria weighting: RANCOM (ranking-comparison method) to derive weightings from ordinal inputs.
  • Policy evaluation & ranking: MARCOS to score alternatives against criteria.
  • Aggregation: Compromise Fuzzy Ranking (CFR) to synthesize model/expert outputs into a group consensus ranking.
• Robustness/risk treatment:
  • Repeated LLM runs and RIDM used to quantify output stability, variability, and decision risk when LLMs are part of the pipeline.
• Evaluation metrics: correspondence between LLM and expert weights; differences in decision matrices and final policy rankings; sensitivity to repeated runs and model choice.

Implications for AI Economics

• Practical role of LLMs in MCDA:
  • High utility for rapid, low-cost elicitation of criteria and initial weighting—can reduce expert time and standardize inputs.
  • Less reliable for full, autonomous generation of decision matrices and final rankings without expert oversight.
• Recommended hybrid workflow:
  • Use LLMs for initial elicitation and RANCOM-based weighting, then have experts validate and adjust decision matrices and MARCOS scores.
  • Use multiple LLMs and ensemble/consensus methods (e.g., CFR) to mitigate single‑model idiosyncrasies.
  • Incorporate repeated runs and RIDM-style risk assessment to detect unstable outputs and quantify decision risk.
• Policy and methodological safeguards:
  • Require transparent prompts, versioning, and documentation of LLM outputs for auditability.
  • Perform sensitivity analyses and expert validation on stages with high LLM–expert divergence.
  • Calibrate or fine-tune models on domain-specific data where possible to reduce variability.
• Research directions for AI economics:
  • Test generalizability across countries, policy domains, and more/broader LLMs.
  • Evaluate cost–benefit tradeoffs of LLM-assisted workflows versus pure expert processes.
  • Develop best-practice protocols for integrating LLMs into multi-criteria policy appraisal, including interpretability and bias assessments.
• Overall: LLMs are promising decision-support components for economic policy appraisal but should be deployed within structured, expert-supervised MCDA pipelines to preserve reliability and legitimacy of policy recommendations.