A new benchmark reveals multimodal models still far short of engineers: MMArch finds the best MLLM scores ~52% while professional experts score ~95%, with failures concentrated in applying principles and composing evidence across figures.

Main Finding

MMArch is a rigorous benchmark that measures multimodal, principle-grounded reasoning in architecture and civil engineering (AEC). Evaluated on 1,212 figure-grounded short-answer items drawn from peer‑reviewed papers, current MLLMs fall far short of human experts: the best proprietary model (GPT‑5.5) reaches ≈51.7% accuracy, the strongest open‑source systems ≈30%, while a domain‑expert human panel scores ≈94.6%. Errors concentrate on applying engineering principles and composing evidence across multiple figures rather than on basic perception, indicating substantial headroom before these models can reliably support professional AEC judgment.

Key Points

• Task focus: each item requires (1) perceiving relevant visual evidence from 1–3 paper figures, (2) identifying the governing architectural/engineering principle, and (3) applying that principle to the evidence to produce a short (≤10‑token) answer.
• Dataset and coverage: 1,212 validated short‑answer items across ten AEC subdomains (structural, seismic, building physics, inspection, heritage, spatial, envelope, urban, BIM/digital, computational design), created from ~10,000 screened papers.
• Construction pipeline to prevent shortcuts:
  • Decoupled planner–writer design: a planner extracts and freezes an answer from the paper text; a separate writer composes a question around the frozen answer.
  • Automated screening: model trials on full input (8), question-only (4), and caption-only (2); items solvable from text/caption alone or too easy are rejected.
  • Blind adversarial audit: three blind agents check evidence visibility, necessity of principle, and search for shortcuts.
  • Dual-path verification: answers independently re-derived via two paths and reconciled.
  • Final expert adjudication: unanimous agreement by three professional architects/engineers required.
• Evaluation protocol:
  • Short‑answer deterministic scoring (unit/synonym/ tolerance normalization) — avoids multiple‑choice inflation and LLM-based judges.
  • 18 models tested (open‑source and proprietary) at temperature 0; single deterministic run per model.
• Main quantitative results:
  • Human experts: ≈94.6% accuracy.
  • Best proprietary models (GPT‑5.5, Claude Opus 4.8): ≈51–52% accuracy.
  • Best open‑source models: ≈30% accuracy (top open models much lower than proprietary).
• Error analysis:
  • Failure types grouped into perception, principle, composition, grounding, and consistency.
  • Composition errors (combining evidence across figures and applying principle) account for the largest share (>~33%).
  • Simple prompting (e.g., chain‑of‑thought) produces small and inconsistent gains—suggesting missing domain knowledge and compositional reasoning capability rather than lack of explicit reasoning traces.
• Resources: dataset, construction code, and benchmark artifacts are publicly available (link in paper).

Data & Methods

• Source material: figures and surrounding context retained from SCI‑indexed journals and arXiv AEC papers; evidence leads (supporting sentences) stored as curator‑only metadata (not exposed to evaluated models).
• Item format: x = (I, Q, A) with I = 1–3 figures, Q = short open-ended question, A = reference short answer.
• Planner–writer pipeline:
  • Planner selects evidence lead and freezes a concise answer (≤10 tokens), plus intermediate observations (for curator use).
  • Writer composes a free-form, varied, and challenging question that does not contain the answer and requires multi-step visual operations.
• Automated screening:
  • Run a “solver” with reasoning disabled over full input (8 trials), question‑only (4 trials), caption‑only (2 trials).
  • Discard items solvable by caption or question alone (text leakage) or that models answer reliably from full input (too easy).
  • Surviving items categorized Hard/Challenge/Deferred for expert review.
• Blind adversarial audit:
  • Three separate auditors (blind to answers and scores) verify evidence visibility, principle necessity, and search adversarially for shortcuts/ambiguities.
• Dual verification:
  • Answers re-derived along two independent paths (paper span vs. image-crop–based computation) and accepted only when reconciled under fixed units/tolerances.
• Expert review:
  • Three professional architects/engineers must unanimously agree that the item is well‑posed, necessary, and correctly answered.
• Final size and yield:
  • From ≈10k screened papers, 1,212 high‑quality items retained (low yield by design to ensure diagnosticity).
• Evaluation specifics:
  • Deterministic decoding (temp=0), max output length 2,048 tokens; single run per model because of determinism.
  • Normalized short‑answer scoring that accounts for units, synonyms, numeric tolerance, accepted alternative answers.

Implications for AI Economics

• Economic potential vs. realism: AEC is a large, high‑value sector where automating parts of the design–analysis–review loop could yield outsized returns (reduced expert time, faster iteration, lower inspection costs). MMArch shows current MLLMs are far from replacing expert judgment for principle‑grounded decisions, so near‑term gains are likely to be in augmentation (assistive tools) rather than substitution.
• R&D investment priorities:
  • High economic leverage from improving compositional multimodal reasoning and domain knowledge grounding (these are the primary bottlenecks identified).
  • Investments in domain‑specific multimodal fine‑tuning, structured knowledge integration (codified engineering rules), and multi‑figure reasoning architectures are likely to have high ROI.
  • Simple prompting or generic scale alone is insufficient; targeted architectural and data strategies are needed.
• Market opportunities:
  • Specialized, domain‑trained MLLMs and toolchains (BIM integration, CAD/figure-aware models, verification pipelines) are promising product areas.
  • Services that combine MLLMs with human experts (expert-in-the-loop workflows, model-assisted review) will likely capture near-term value while mitigating liability.
• Procurement, regulation, and standards:
  • Benchmarks like MMArch should be part of procurement and regulatory evaluation to avoid overclaiming capabilities; vendors should be required to report performance on domain‑specific, shortcut‑resistant tests.
  • Deterministic short‑answer testing with provenance and anti‑shortcut construction is a model for credible capability claims in safety‑critical domains.
• Labor and liability:
  • Given the ~40‑point gap to experts, deploying MLLMs without human oversight in safety‑critical design/inspection tasks would be premature and risky.
  • Economic impacts will be more about productivity augmentation (reducing routine tasks, pre‑screening, drafting reports) than headcount reduction in the medium term.
• Policy and funding:
  • Public and private funders should prioritize research that closes the compositional and principle‑grounding gaps (multimodal reasoning, verified grounding, cross‑figure synthesis), because the societal and economic gains from reliable automation in AEC could be large.
• Cross‑domain benchmarking lessons:
  • The MMArch construction methodology (planner–writer decoupling, anti‑shortcut screening, blind adversarial audit, dual verification, expert unanimity) is a useful template for economically important domains where incorrect automation carries high cost (medicine, finance, infrastructure).

Summary: MMArch provides a rigorous, domain‑specific measure of multimodal reasoning under engineering principles. Its results show substantial capability gaps in current MLLMs—especially in composing multiple pieces of evidence and applying domain rules—which directs R&D and investment toward compositional multimodal models, domain grounding, and hybrid human–AI workflows if we want economically meaningful automation in AEC.