Charts help multimodal LLMs diagnose problems, forecast risk and write board rationales — but they consistently break budget allocation decisions; indiscriminate visual augmentation can improve evidence use while degrading constrained, actionable choices.

Main Finding

Multimodal LLMs (MLLMs) improve evidence-centric executive reasoning (diagnosis, prioritization, risk forecasting, and board justification) but systematically degrade constrained resource-allocation decisions. The authors introduce C-SUITEBENCH — a paired text-only vs. multimodal benchmark — and identify a “multimodal integration paradox”: visual inputs increase grounding of evidence but harm constraint satisfaction and strategic fit when models must output numerically constrained allocations.

Key Points

• Benchmark and setup
  • C-SUITEBENCH: 50 structured company scenarios × 5 executive tasks (250 instances per input mode), paired text-only and multimodal versions (visuals are rendered from the same underlying data).
  • Tasks: T1 Organizational diagnosis, T2 Evidence prioritization, T3 Resource reallocation (numerical budget allocations under constraints), T4 Risk forecasting, T5 Board justification.
  • Evaluated 9 frontier MLLMs (e.g., GPT-5.4 mini/nano, Claude Opus 4.8, Qwen2.5-VL, Gemini 3 Flash, several MiniMax variants, Mistral Small). Prompts identical across modes; images provided inline in multimodal runs.
• Aggregate multimodal effects (mean multimodal uplift Δ = multimodal − text-only):
  • T1 Diagnose: +0.07
  • T2 Prioritize: +0.10
  • T3 Allocate (constrained reallocation): −0.08 (negative for all nine models)
  • T4 Forecast: +0.24 (largest uplift)
  • T5 Justify: +0.17
• The paradoxical finding
  • Visual grounding proxy increased substantially under multimodal input (models attend to/reflect visual signals better).
  • Despite better grounding, constraint validity (adherence to numeric allocation constraints) and strategic fit (alignment with priorities) declined for T3.
  • Authors attribute this to signal crowding: multiple visual channels add competing numeric signals (unit KPIs, headroom, rate changes) that interfere with satisfying explicit decoding constraints.
• Ablations
  • Finance-only, growth-only, and ops-only ablations show each visual channel individually tends to help performance, but combining channels (full multimodal) degrades constrained allocation consistently. This suggests interactions among visual channels — not a single channel — produce the failure mode.
• Robustness
  • The T3 degradation was consistent across model families and scenario difficulty families (Easy, Fragile, Tension, Adversarial), with larger drops in Tension/Adversarial cases.

Data & Methods

• Controlled paired design: identical underlying decision problems; difference is presence/absence of rendered visuals (charts, KPIs, dashboards).
• Scenario space: 50 mid-sized enterprise scenarios crafted to surface visual signals that are hard to recover from text alone; four difficulty families (Easy, Fragile, Tension, Adversarial).
• Evaluation: task-specific automatic scorers with tailored rule/overlap/constraint-based dimensions, aggregated into scores in [0,1]. Multimodal uplift Δ = score_multimodal − score_text.
• Models: nine contemporary multimodal-capable LLMs; no model-specific prompt tuning.
• Visual ablations: evaluate effects of providing only one visual category (finance / growth / ops) versus all together.
• Diagnostics: sub-dimension analyses (visual grounding proxy, constraint validity, strategic fit, leading-indicator recall, evidence citation/tradeoff coverage).

Implications for AI Economics

• For adoption and productivity modeling
  • Heterogeneous benefits: Multimodal agents add value where decisions are evidence-centric and interpretative (diagnosis, forecasting, communication). Productivity gains from multimodal AI will likely be task- and stage-specific within firms.
  • Hidden costs for constrained decisions: Where decisions require strict numeric feasibility (budgets, resource plans), naive multimodal augmentation can reduce decision quality. Economic estimates of automation benefits must account for such asymmetric effects.
• On labor substitution vs. complementarity
  • Complementarity likely remains high for evidence-centric tasks (visual-augmented forecasting, report drafting). For constrained allocation tasks, human oversight (or specialized algorithmic layers enforcing constraints) will remain complementary, delaying full substitution.
• Organizational design & incentives
  • Firms should not indiscriminately expose decision agents to all available visual data. Selective grounding and interface design (choose channels strategically) can yield better downstream decisions.
  • Procurement and governance: procurement decisions for MLLM tools should evaluate multimodal performance on the full decision pipeline, not just perceptual benchmarks.
• Model design and vendor implications
  • Need for architectures and decoding strategies that separate perception (visual grounding) from constrained action (hard constraints enforcement). E.g., constraint-aware decoders, explicit optimization back-ends, or post-hoc projection of outputs onto feasibility sets.
  • Vendors should provide mode-specific safety/constraint mechanisms and diagnostics showing whether multimodal inputs affect feasibility of outputs.
• Policy, risk, and liability
  • Regulatory assessments of AI for executive use should examine not just interpretability/grounding but also whether outputs meet hard constraints. Failures in constrained actions pose financial and legal risks.
  • Disclosure requirements: whether and which visual data were used in automated executive decisions, especially for investor-facing outputs.
• Directions for economic research
  • Empirical work: measure firm-level productivity changes from multimodal AI by task type; estimate how much value accrues to forecasting/communication vs. action-execution tasks.
  • Modeling: build structural models where multimodal information improves signal quality but may raise decision error in constrained choice problems — analyze equilibrium adoption and complementary investments (e.g., constraint-enforcement software, human oversight).
  • Cost-benefit frameworks: include costs of additional errors in constrained decisions, compliance risk, and mitigation (human review, specialized modules).
• Practical recommendations for firms
  • Use multimodal MLLMs for evidence extraction, forecasting, and narrative generation; require human sign-off for numerically constrained allocations or integrate explicit constraint-enforcement modules.
  • When deploying, instrument outputs with constraint-validity checks and ablate visual channels as needed (sometimes single-channel visuals help while full multimodal harms).
  • Evaluate vendors on paired text-only vs. multimodal tasks that reflect real organizational constraints.

Limitations to note - Benchmark scenarios are controlled and synthetic; external validity to complex real-world corporate data needs further testing. - Automatic scorers may imperfectly capture nuanced strategic fit; human evaluation could complement these metrics. - The failure mode is tied to current generation MLLMs and decoding practices; future models or architectures with explicit optimization layers may mitigate the paradox.

If you want, I can: - Extract model-level deltas and rank models by multimodal improvements per task. - Suggest concrete architectures or prompt+postprocessing patterns to mitigate the T3 failure (e.g., constraint projection, two-stage pipelines).