Two large language models diverge: DeepSeek-V3 is more accurate and stable at predicting firms' earnings direction, while GPT-4 gives superior explanatory insight and an edge during unstable recovery periods; combining both approaches appears to offer the best decision‑support trade-off.

Main Finding

When given five years of historical financial statements (2017–2021) to predict earnings direction for 2022–2024, GPT‑4 (ChatGPT) produced richer, more interpretable justifications and performed relatively better in unstable or recovery scenarios, while DeepSeek‑V3 produced more stable and more accurate directional forecasts in stable, regulation‑driven environments. The paper emphasizes a trade‑off between interpretability (ChatGPT) and reliability/stability (DeepSeek) and recommends hybrid approaches for practical decision support. These results challenge strict forms of the Efficient Market Hypothesis by showing systematic, model-dependent predictive differences from identical inputs.

Key Points

• Models compared: OpenAI GPT‑4 (ChatGPT) vs DeepSeek‑V3‑Chat.
• Task: directional earnings forecast (Increase / Decrease / Stable) plus approximate % change and a short justification, using only audited financial statements (2017–2021).
• Sample: 15 publicly listed firms across three regions (Turkey, Europe, USA) and three sectors (banking, technology, industrial). Examples: Apple, Amazon, JPMorgan, BNP Paribas, Nestlé, Volkswagen, Arçelik, Tüpraş, Garanti BBVA, Ziraat, Halkbank.
• Input features: Total Revenue, EPS, Net Income, Total Assets, Total Liabilities, Shareholders’ Equity, EBIT, Current Ratio (provided in Excel).
• Currency handling: analyses done in firms’ local reporting currencies (no cross‑firm currency conversion); forecasts used intra‑firm year‑over‑year percentage changes to avoid nominal distortion.
• Prompt engineering: standardized template, chain‑of‑thought (CoT) prompting, constrained to financial statement data only, structured tabular output required. Separate sessions per model-company-year.
• Evaluation metrics: direction accuracy, confusion matrices, bias/consistency metrics, and an interpretive “insight quality” score (qualitative scoring of explanations).
• Principal empirical finding: ChatGPT excels in generating plausible reasoning and handles volatile/recovery cases better; DeepSeek is more accurate and consistent in well‑regulated, stable environments (notably some banking contexts).
• Interpretation: results highlight model‑specific heuristic choices and training biases; neither model dominates across all sectors/regions.
• Recommendation: hybrid systems that combine an interpretable LLM for rationale and a sturdier/statistically calibrated model for final directional decision provide improved decision support.

Data & Methods

• Design: comparative longitudinal forecasting exercise. Models used only firm financial statements (2017–2021) to predict direction of earnings for 2022–2024.
• Sample selection criteria: asset size, exchange listing, data availability; 15 firms across Turkey, Europe, USA in banking, technology, industrial/energy.
• Financial statements used: Income Statement, Balance Sheet, Cash Flow, Statement of Changes in Equity. Data sources: KAP (Turkey), official annual reports (Europe, IFRS), investor relations/AnnualReports.com (USA, US GAAP).
• Feature engineering / derived metrics:
  • EPS (IAS 33 / TFRS 33 methods when not directly reported).
  • EBIT: computed differently for corporates (operating income + interest expense) and banks (approximated as net interest income + non‑interest income − operating expenses or operating profit before provisions & taxes).
  • Current ratio and standard balance‑sheet aggregates derived per Penman/Damodaran conventions.
• Prompting protocol:
  • Unified “forget previous instructions” template making the model act as a financial expert.
  • Explicit constraints: use only provided financials; produce year‑by‑year direction, % estimate, and ≤3 sentence justification.
  • Chain‑of‑thought encouraged to increase reasoning traceability.
• Model execution: identical inputs and prompt format supplied in separate sessions to GPT‑4 and DeepSeek‑V3.
• Evaluation:
  • Directional predictions compared to realized outcomes for 2022–2024.
  • Metrics: accuracy, class confusion matrices (false positives/negatives per class), consistency over years, bias tendencies, and a qualitative interpretive score for explanations.
• Limitations acknowledged by authors:
  • Small sample (15 firms) limiting generalizability.
  • Exclusion of macro/market sentiment and non‑financial textual sources.
  • No currency normalization across firms limits cross‑firm magnitude comparison (authors argue intra‑firm % changes mitigate this).
  • Dependence on prompt design, model versions, and possible hallucinations or table‑reading errors.
  • Tests are retrospective; not a live trading or real‑time decision experiment.

Implications for AI Economics

• For theory: systematic, model‑dependent predictive differences using identical inputs present an empirical challenge to strict EMH claims that all useful information is already fully and equivalently priced; model processing matters.
• For practice (analysts / institutions):
  • Match model choice to environment: use more interpretable LLMs for volatile/recovery situations where nuanced reasoning aids judgment; prefer more stable/accurate models (or calibrated ensembling) in regulated, stable sectors (some banking contexts).
  • Adopt hybrid decision‑support workflows: combine LLM rationale (for auditability and human review) with a calibrated predictive engine for final directional calls.
  • Insist on prompt standardization, CoT-style explanations, and version control for reproducibility and audit trails.
• For model developers:
  • Improve sectoral calibration (accounting standard diversity, bank vs corporate accounting), table understanding, and robustness to input formatting.
  • Provide tools for quantitative calibration of LLM probability outputs (to reduce over/underconfidence) and for structured explainability metrics to support auditability in finance.
• For regulators and auditors:
  • Require transparency around prompt and model versions used in financial decision contexts; require backtesting and documentation of model performance across regions/sectors.
  • Encourage standards for interpretability scoring and stress‑testing LLMs on accounting standards heterogeneity (IFRS vs US GAAP vs local TFRS).
• For research:
  • Scale up sample sizes, include macro/market sentiment and textual disclosures, and benchmark against human analysts and specialized predictive ML models.
  • Explore formal ensemble/hybrid strategies and calibration methods that reconcile interpretability and directional accuracy.
  • Test real‑time deployment, economic value (e.g., trading or risk management performance), and fairness/ethical considerations in automated forecasting.

Overall, the paper provides a focused, controlled comparison that surfaces meaningful trade‑offs (interpretability vs reliability) when using general‑purpose LLMs for financial forecasting and makes a strong case for hybrid, audited decision‑support systems tailored by sector and regime.