Corporate audio, images and video remain an untapped source of economic information for accounting research; integrating multimodal AI with financial text and numbers could improve measurement, detect misreporting and change how markets process disclosures, but doing so requires careful causal methods, benchmarks, and regulatory safeguards.

Main Finding

Multimodal (visual, audio, video, and their interactions with text and numbers) financial information is substantially underutilized in accounting research. The paper surveys existing work, outlines theoretical and technical foundations, and proposes a research agenda centered on three roles for multimodal information fusion in accounting: improving information completeness, enabling verification, and enhancing effectiveness of disclosures and analyses.

Key Points

• Current focus: Accounting and finance research predominantly uses numerical and textual data; visual, audio, video modalities and cross-modal interactions are comparatively neglected.
• Opportunity: Increasing availability of multimodal data (e.g., earnings-call audio, presentation slides, product images, CEO videos, social media posts) and advances in AI (multimodal representation learning, transformers) enable richer analyses of corporate signals.
• Three proposed roles for multimodal fusion:
  • Information completeness — combining modalities fills gaps in what single modes convey (e.g., images or video reveal product quality or production scale not captured in numbers).
  • Verification — cross-checking information across modalities helps detect misreporting, inconsistencies, or fraud (e.g., mismatch between corporate claims and visual evidence).
  • Effectiveness — multimodal presentation affects how information is perceived and acted upon by investors and stakeholders (e.g., tone in audio/video influences market reactions beyond textual content).
• Technical foundations: multimodal fusion strategies (early/intermediate/late), pretraining and transfer learning, alignment and synchronization of signals, handling missing/noisy modalities, and interpretability methods.
• Challenges and risks: data collection/labeling, privacy and proprietary constraints, heterogeneity of modalities, measurement error, causal identification, model explainability, regulatory implications, and potential for adversarial manipulation or bias amplification.

Data & Methods

• Typical data sources:
  • Corporate disclosures: annual reports, MD&A (text + images), slides, management presentations, video/webcast recordings, earnings-call transcripts and audio.
  • External signals: product images, satellite imagery, store-front photos, social media posts, analyst presentations, job postings, press coverage.
  • Audit and compliance artifacts where available.
• Methods surveyed / recommended:
  • Representation learning: multimodal embeddings, contrastive learning, joint and cross-modal encoders.
  • Fusion strategies: early (concatenate features), intermediate (cross-attention), late (ensemble predictions); model choice depends on alignment and task.
  • Specialized models: vision-language models (VLMs), speech-to-text + prosody analysis, multimodal transformers, video understanding networks.
  • Classical + causal methods: difference-in-differences, event studies, instrumental variables, and experiments to establish causal mechanisms (e.g., how multimodal cues affect investor behavior).
  • Robustness and interpretability: saliency/attention visualization, counterfactual explanations, human-in-the-loop validation.
  • Practical considerations: handling missing modalities, synchronizing time-series multimodal streams, annotation protocols, privacy-preserving techniques.
• Evaluation:
  • Task-specific metrics (classification accuracy, AUC, forecasting error) and economic metrics (abnormal returns, trading volume, cost of capital, audit outcomes).
  • Benchmark datasets and reproducibility practices are emphasized as needed.

Implications for AI Economics

• Measurement and information environment:
  • Multimodal signals can improve measurement of firm fundamentals and soft information, reducing measurement error and enabling richer tests of asset pricing and information asymmetry theories.
  • Better proxies for unobserved firm activities (e.g., production scale via imagery) can refine empirical estimates of productivity, investment, and risk.
• Market efficiency and asset pricing:
  • Adding multimodal cues may alter how quickly and accurately markets incorporate information; models that exploit multimodal data could generate predictability that challenges semi-strong efficiency tests.
  • Incorporating multimodal features into risk models and event studies could change inferred risk premia and cross-sectional return patterns.
• Corporate behavior and disclosure strategy:
  • Firms may strategically deploy multimodal content (visuals, tone) to influence investor perceptions; regulators and researchers should study detectability and materiality of such strategies.
  • Multimodal audit and verification tools could improve audit quality and fraud detection, but also raise questions about standards and liability.
• Policy, regulation, and ethics:
  • Use of multimodal AI raises privacy, consent, and manipulation concerns (e.g., deepfakes); regulators must consider guidelines for disclosure formats and verification.
  • Standardization of multimodal reporting and data-sharing practices would facilitate research and market transparency.
• Labor and market structure:
  • Demand for new skills (multimodal analytics) in accounting and finance professions will grow; adoption may change the role of human analysts toward oversight and interpretation.
• Research agenda recommendations:
  • Build benchmark multimodal datasets with clear provenance, labeling, and privacy safeguards.
  • Combine causal identification strategies with multimodal ML to move beyond prediction to explanation.
  • Invest in interpretability and robustness checks to ensure economic validity and guard against manipulation.
  • Foster interdisciplinary collaboration (accounting, computer vision, NLP, econometrics, regulation) and open reproducible code/data to accelerate progress.

Short takeaway: Multimodal AI offers substantial potential to improve accounting measurement, verification, and the study of information effects in markets, but realizing this potential requires careful methodological, causal, ethical, and regulatory work.