0 cumulative citations
View corpus contextAI systems can sketch believable CAD but struggle to make engineering-grade parts: a new benchmark shows many models produce executable-looking geometry that violates engineering, parametric, or physics tests, while editing existing CAD is markedly easier than generating correct parametric designs; assembly predictions commonly locate the right region but miss the exact mating entities.
CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation
CADEngBench demonstrates that modern multimodal code-capable models often produce visually plausible CAD but frequently fail engineering-grade checks—parametric integrity, matched FEA, and exact assembly grounding—while controlled editing of supplied CAD is substantially easier than correct generation from scratch.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
A CAD model is not engineering-grade merely because it looks correct. It must satisfy design requirements, respond predictably to parameter changes, support controlled edits, match a reference structural response under a declared analysis, and connect to other parts through valid joints. We present CADEngBench, a two-track benchmark for these capabilities. CADEngBench-P evaluates 300 parametric parts, each used for one zero-to-CAD task and one functional-editing task (600 tasks in total), through boundary-representation (B-Rep) validity, engineering and DFM checks, parameter-family perturbations, functional editing, and matched linear-static FEA in CalculiX. CADEngBench-A evaluates 150 body pairs through ranked joint retrieval, exact face-and-edge grounding, joint-frame prediction, and kinematic verification. Across eight multimodal, code-capable models, editing supplied CAD is substantially easier than generating it, while complex edits and matched FEA remain difficult. Assembly predictions often locate the relevant region but fail to recover the recorded joint or mating entities. These results show that CAD evaluation must test engineering behavior rather than appearance alone.
Summary
Main Finding
CADEngBench introduces a rigorous, engineering-focused benchmark for parametric CAD, assembly grounding, and physics verification. Across 8 leading multimodal/code-capable models, outputs that merely "look like" CAD often fail when judged as executable engineering artifacts: editing supplied CAD is substantially easier than generating correct parametric CAD from scratch, matched FEA and exact assembly grounding remain difficult, and many apparently valid generations fail engineering/DFM checks or parametric integrity.
Key Points
- Benchmark structure
- Two complementary tracks:
- CADEngBench-P (parts): 300 parts → 600 tasks (zero-to-CAD generation + functional editing). Layered checks L0–L3 (execution → engineering/DFM → parametric perturbation/editing → matched FEA).
- CADEngBench-A (assembly): 150 body pairs (120 used for reported results) with strict B-Rep entity grounding, joint-family retrieval, frame prediction, and kinematic motion checks (A0–A2).
- Two complementary tracks:
- Layered evaluation emphasizes engineering behavior, not appearance:
- L0: CadQuery program runs, creates a valid B-Rep, and exports/imports STEP.
- L1: Engineering/DFM checks (dimensions, feature counts, min wall/hole rules).
- L2-Z: Parametric integrity — verify declared parameter controls intended geometry across multiple values.
- L2-E: Functional editing — perform specified edit while preserving protected geometry.
- L3: Matched linear-static FEA (CalculiX) comparing stress/displacement/compliance across parameter families.
- A1: Exact face/edge retrieval and joint-family prediction (ranked).
- A2: Joint frame prediction + kinematic verification in PyBullet (end-to-end).
- Dataset and tooling
- CADEngBench-P: 300 items (159 BenchCAD CadQuery programs, 141 Fusion360 reconstructions), 102 families, 55 CAD ops, 12 edit types; 164 items qualify for L3 FEA testing.
- CADEngBench-A: 150 body pairs decomposed into labeled B-Rep faces/edges; recorded joints grounded to entities and frames.
- Tools: CadQuery, Fusion operation histories, Gmsh meshing, CalculiX FEA, PyBullet kinematics.
- Empirical summary (selected results)
- Executability (L0): 1,030/2,400 generated programs passed L0 (~43%).
- Engineering compliance (L1): Only 432 of those L0-pass outputs also passed L1 → ~58% of executable outputs violated at least one engineering or DFM requirement.
- Editing vs generation: Among scoreable pairs, 4.64× more cases passed only editing than passed only generation (editing is much easier).
- FEA: 164 FEA-eligible parts; models had limited reach to produce valid matched simulations, and many generated designs diverged in stress/response when sweep of parameter states was applied (example failure: min-radius stress 3.43 MPa vs reference 1.15 MPa).
- Assembly: Predictions often localized correct region but failed to recover exact mating entities; A2 end-to-end pass rates were low. Cross-track correlation between parametric construction and assembly motion performance is near zero (Spearman ρs ≈ 0.05), implying different capabilities are required.
- Model coverage: Evaluated eight state-of-the-art multimodal/code-capable models (GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, GLM-4.6V, Kimi K2.5, Mistral Medium 3.5, Llama 4 Maverick, Qwen3.5-35B-A3B). No single model dominated all evaluation layers.
Data & Methods
- Task construction
- Replayed CadQuery programs and Fusion360 histories to recover executable operation histories, parameters, and B-Reps.
- Deterministic extraction of parameters, protected invariants, required interfaces, and FEA boundary selections from source metadata where available.
- Evaluation methodology
- Single-response per task from each model; no retries allowed.
- L0–L2 evaluated on all 300 items where applicable; L3 evaluated on 164 items qualified for linear-static FEA comparison.
- CADEngBench-A: 150 body pairs (30 used for development), remaining 120 items reported; faces/edges indexed with stable IDs and ground-truth joint data withheld from model input.
- Kinematic verification: predicted frames converted to URDF joints and validated in PyBullet (allowed DOFs and forbidden motions tested).
- Physics comparison (L3)
- For each parameter state, both reference and generated solids were meshed with quadratic tetrahedra (Gmsh) and solved in CalculiX.
- Metrics compared via multiplicative log differences of quantities (95th-percentile von Mises stress, max displacement, normalized compliance or stress concentration). Worst-case across parameter family determines pass/fail tolerance.
- Metrics & statistics
- L0/L1/L2/L3 pass rates, reach (ability to run L3), pair-pass (both valid sim and within tolerance), Entity@k, Typed@k, MRR, A2 end-to-end pass.
- Confidence intervals via 10,000 item-clustered bootstrap samples; pairwise comparisons use McNemar tests with Holm correction.
Implications for AI Economics
- Realized productivity gains will be less than naive appearance-based estimates
- Many generated CAD outputs currently fail engineering checks or parametric integrity; time saved by automating "drafting" or visual geometry may be offset by verification and rework costs.
- The observed advantage of editing over generation suggests near-term productivity improvements are likelier when AI augments human engineers (assistive editing, param adjustments) rather than replacing end-to-end CAD creation.
- Validation and verification costs become first-order economic factors
- Because matched FEA, exact mating entity selection, and parametric correctness are nontrivial, firms adopting AI CAD tools will need investment in verification infrastructure (meshing/FEA, unit checks, human QA). Those costs reduce net labor substitution and shift savings from broad headcount reductions to task reallocation and faster iteration cycles.
- Market differentiation and willingness-to-pay
- Benchmarks like CADEngBench create a measurable product quality axis. Enterprise buyers will likely pay premiums for models/systems that can demonstrably satisfy higher L-levels (L1–L3) and assembly grounding—this can stratify vendors and create certification/benchmark-based procurement.
- Strategic R&D and capital allocation
- Economic incentives will favor models that integrate symbolic/executable CAD generation with physics engines and deterministic compilers (closing the loop between generation and solver-based verification).
- Startups and incumbents may invest in verticalized stacks (CAD front-end + internal solver + dataset of validated parametric families) rather than general-purpose LLMs, because engineering-grade outputs require deterministic, testable execution semantics.
- Labor and skills implications
- Near-term: increased demand for roles that validate, interpret, and harden AI-generated CAD (simulation engineers, CAD-verification specialists).
- Medium-term: shift of human designers toward higher-level specification, systems integration, and verification-centric work; routine geometric modeling tasks may be semi-automated.
- Liability, standards, and regulation
- Failures in engineering-grade outputs (e.g., wrong parameter coupling, incorrect stress response) create safety and liability risks. Emergence of standards/certification for "engineering-grade AI CAD" (akin to ISO/industry standards) is likely; firms may be required to demonstrate model passes on benchmarks like CADEngBench for regulated products.
- Impact on innovation and market dynamics
- Better tooling for rapid prototyping (editing + verified parametric sweeps) could accelerate product iteration and lower the effective cost of experimentation, boosting innovation in SMEs that cannot hire large CAD teams.
- Conversely, the need for verification may concentrate value in firms that can afford the integrated stacks and certified models, potentially increasing returns to scale and some consolidation.
- Measuring economic value
- Return-on-investment for AI CAD adoption must factor: model subscription/licensing, integration and validation tooling, time to reliable output (including verification loops), and residual human oversight. Benchmarks that expose hidden brittleness (CADEngBench does) enable more realistic ROI modeling.
Takeaway for AI economics stakeholders: CADEngBench reveals that "automation" in CAD should be economically scoped by engineering correctness and verifiability, not by surface realism alone. Contracts, procurement, and R&D strategies should price in verification costs and favor solutions that demonstrably satisfy layered engineering checks (especially parametric integrity and matched physics), since those determine actual labor substitution, liability exposure, and adoption value.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents a comprehensive, reproducible benchmark and evaluates eight leading multimodal/code-capable models across many tasks with clear, deterministic checks (L0–L3, A0–A2). Results convincingly show relative strengths and failure modes within the dataset. However, claims about broader AI capability or real-world engineering impact are limited by the dataset composition (BenchCAD + Fusion 360), the selected models, and the specific evaluation thresholds and linear-static FEA regime, so external generalization is not established.
Methods Rigorhigh — The benchmark is carefully constructed with deterministic replay of histories, explicit B-Rep grounding, layer-wise evaluation (ranging from executability to matched FEA and kinematic verification), clear success/failure criteria, statistical procedures (clustered bootstrap, Holm-corrected tests), and no-retry protocol for models; these elements indicate strong internal validity and reproducibility.
SampleCADEngBench-P: 300 parts (159 BenchCAD CadQuery programs + 141 Fusion 360 reconstruction items), producing 600 tasks (zero-to-CAD + functional editing). 100 easy/medium/hard parts, 102 BenchCAD families, 55 CAD operations; 164 parts have matched L3 linear-static FEA tests across parameter families (including axial tension, cantilever, restrained acceleration). CADEngBench-A: 150 body pairs drawn from Fusion 360 assembly-joint data (120 items used for reported comparisons after excluding development set), covering 7 joint families and ~17k B-Rep candidates. Evaluated eight multimodal, code-capable models (GPT-5.2, Claude 4.5, Gemini 3 Flash, GLM-4.6V, Kimi K2.5, Mistral Medium 3.5, Llama 4 Maverick, Qwen3.5-35B-A3B).
Themeshuman_ai_collab productivity
GeneralizabilityDataset limited to BenchCAD and Fusion 360 artifacts; other CAD ecosystems, proprietary industrial part libraries, or different design conventions may behave differently, FEA evaluation restricted to linear-static analyses and specific load families; nonlinear, dynamic, or multi-physics behavior is not assessed, Evaluated models represent a snapshot of systems available in 2025–26; future or domain-specialist models may perform differently, CadQuery/STEP-centered workflow biases results toward code-capable, programmatic CAD generation; GUI-driven or feature-based CAD systems could present different challenges, Evaluation thresholds (e.g., 1%/5° frame matching, DFM minima) and hidden checks shape pass rates and may not align with every industrial tolerance or use case
Claims (12)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| CADEngBench evaluates CAD using two tracks: CADEngBench-P contains 300 parametric parts used for 600 generation and editing tasks, while CADEngBench-A contains 150 assembly body pairs. Other | positive | Benchmark coverage of parametric CAD, editing, and assembly reasoning capabilities |
Reading fidelity
high
Study strength
high
|
n=450
600 tasks and 150 body pairs
|
| Only 1,030 of 2,400 generated programs passed the basic executability and valid-solid check (L0), and only 432 of those also passed engineering and DFM checks (L1). Output Quality | negative | Executable CAD generation and satisfaction of engineering/DFM requirements |
Reading fidelity
high
Study strength
high
|
n=2400
1,030/2,400 pass L0; 432 pass both L0 and L1
|
| Among executable generated outputs, 58.1% violated at least one stated engineering or DFM requirement. Output Quality | negative | Engineering and design-for-manufacturing requirement satisfaction |
Reading fidelity
high
Study strength
high
|
n=1030
58.1%
|
| Editing supplied CAD was substantially more successful than generating CAD from scratch: among 2,330 comparable item-model pairs, 1,071 passed editing only, compared with 231 that passed generation only. Output Quality | positive | Functional CAD-editing success relative to zero-to-CAD generation success |
Reading fidelity
high
Study strength
high
|
n=2330
4.64× edit-only asymmetry
|
| The relative difficulty of editing versus generation varied by edit structure rather than being uniform across all functional edits. Output Quality | mixed | Functional-editing success across different edit types |
Reading fidelity
high
Study strength
medium
|
n=2330
|
| No evaluated model performed best across all benchmark stages. Output Quality | mixed | Model performance across CAD generation, editing, physics verification, and assembly reasoning |
Reading fidelity
high
Study strength
medium
|
n=8
|
| Model rankings for generation and editing were only weakly associated, with a Spearman rank correlation of 0.214. Output Quality | mixed | Association between model performance rankings for CAD generation and editing |
Reading fidelity
high
Study strength
medium
|
n=8
ρs = 0.214
|
| Parametric construction and assembly-motion performance were nearly unrelated across models, with a Spearman rank correlation of 0.048. Output Quality | null_result | Association between parametric CAD construction and assembly-motion reasoning performance |
Reading fidelity
high
Study strength
medium
|
n=8
ρs = 0.048
|
| Gemini 3 Flash achieved a 46.3% matched-FEA pair-pass rate, passing 119 of 257 evaluated matched-FEA cases. Output Quality | positive | Agreement between generated and reference structural responses under matched FEA |
Reading fidelity
high
Study strength
medium
|
n=257
46.3% (119/257)
|
| Claude 4.5 had the highest reported L3 simulation reach, with 54.3% of the 164 eligible parts reaching the matched-FEA evaluation stage. Output Quality | positive | Successful execution/reach of matched FEA evaluation |
Reading fidelity
high
Study strength
medium
|
n=164
54.3% (89/164)
|
| Assembly predictions frequently identified a plausible region but failed to recover the exact recorded joint or mating entities. Output Quality | negative | Exact joint-family and mating-face/edge retrieval in assembly reasoning |
Reading fidelity
high
Study strength
medium
|
n=120
|
| A generated parametric program can pass individual FEA states while failing across a parameter family because of unintended parameter coupling. Output Quality | negative | Physics consistency across parameter perturbations |
Reading fidelity
high
Study strength
medium
|
n=1
3.43 versus 1.15 MPa at minimum radius
|