Leading LLMs disagree about which startups to back: GPT-4o, Claude 3.5 Sonnet and DeepSeek-V2 yield systematically different evaluation scores, funding recommendations and confidence levels on the same 20 pitch decks, and show wide variation in reliability (ICC range 0.24–0.93), implying model choice can materially shape investment outcomes.

Main Finding

Different large language models (GPT-4o, Claude 3.5 Sonnet, DeepSeek‑V2) produce systematically different investment evaluations in a controlled venture‑screening simulation. Differences are statistically significant and reproducible across repeated runs: models vary not only in average recommendations (funding amounts, scores, expressed confidence) but also in the stability of those recommendations (ICC reliability 0.240–0.930). Three distinct "financial personalities" emerge — cautious allocator (GPT‑4o), conservative scorer (DeepSeek‑V2), and narrative funder (Claude 3.5 Sonnet) — implying model choice materially shapes funding outcomes.

Key Points

• Experimental evidence contradicts the “interchangeable LLM” assumption: model choice materially affects high‑stakes outputs.
• Models differ along multiple margins: funding recommendation, evaluation scores, sensitivity to qualitative narrative, and expressed confidence.
• Reliability matters: some models produce stable judgments across repeated runs (high ICC), others are noisy (low ICC). Reported ICC range: 0.240–0.930.
• Behavioral archetypes identified:
  • GPT‑4o: favorable evaluations but conservative funding decisions — “cautious allocator.”
  • DeepSeek‑V2: stringent, highly consistent scoring but systematically lower funding — “conservative scorer.”
  • Claude 3.5 Sonnet: responsive to qualitative aspects, somewhat higher funding, strong cross‑run reliability — “narrative funder.”
• Practical implication: average performance metrics alone are insufficient; operational stability and behavioral tendencies should be incorporated into model selection.
• Caveat: sample is exploratory (20 real pitch decks); findings are suggestive rather than definitive.

Data & Methods

• Design: simulation‑based experimental study using authentic startup pitch decks to preserve external validity while enabling controlled comparisons.
• Models compared: GPT‑4o, Claude 3.5 Sonnet, DeepSeek‑V2.
• Stimuli: 20 real startup pitch decks spanning industries and funding stages.
• Repeats to separate noise from persistent tendencies: models were evaluated multiple times under identical conditions (five repeated runs reported to account for stochastic output variation).
• Outputs collected: funding recommendation (likely numeric / binary), overall evaluation scores, expressed confidence, qualitative comments.
• Analysis:
  • Statistical comparison across models to test systematic differences in recommendations and scores.
  • Reliability quantified via intraclass correlation coefficients (ICC) to measure stability across repeated runs (reported ICCs between 0.240 and 0.930).
• Interpretation framed through a proposed “financial personality” framework that maps behavioral tendencies to archetypal profiles.

Implications for AI Economics

• Model non‑neutrality can alter capital allocation: choosing different LLMs can systematically change which startups receive funding and how much, potentially affecting which technologies and firms scale.
• Selection of foundation models is a strategic economic decision, not a purely technical procurement choice—firms should match model behavioral profiles to their organizational objectives (e.g., risk tolerance, desire for narrative sensitivity).
• Market‑level effects: if many investors adopt the same model, its embedded biases/preferences could amplify selection effects and reduce diversity in funded ideas, shaping innovation trajectories.
• Evaluation metrics for procurement and regulation should include reliability/stability (ICC or similar) alongside average accuracy/performance measures.
• Governance and procurement recommendations:
  • Benchmark models on task‑specific behavioral tendencies, not just standard ML metrics.
  • Use reliability testing (repeated runs) to understand stochasticity and persistence of behaviors.
  • Consider ensembles or model mixes to diversify behavioral tendencies and mitigate single‑model distortions.
  • Implement auditing, monitoring, and human‑in‑the‑loop checks for high‑stakes decisions.
• Research and policy priorities: expand empirical evidence (larger sample sizes, more models, field deployments), quantify long‑run effects of model‑driven allocation on innovation ecosystems, and consider disclosure requirements for model‑driven decision systems in financial contexts.

Limitations to bear in mind: exploratory study with 20 pitch decks; results indicate sizable effects but require larger, multi‑setting replication before strong generalization.