0 cumulative citations
View corpus contextA specialized vision-agent system automates residential blueprint material takeoffs to near-human completeness and precision: Handoff-H1 scores 81.6% on a 10-project benchmark—roughly 20 points above the best generalist agent and modestly above professional estimators—by combining tailored vision models, a curated construction knowledge base, and multi-stage agent orchestration.
Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprints
Handoff-H1, a three-layer vision-agent system combining specialized CV models, a curated construction knowledge base, and orchestrated agents, achieves an 81.6% composite score on a 10-project residential blueprint takeoff benchmark—outperforming seven frontier agent baselines and slightly exceeding independent professional estimators on composite metrics.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Converting a set of architectural blueprints into a complete material quantity takeoff requires visual perception across drawing sheets, dimensional and multi-hop reasoning, and grounding in construction conventions that the drawings never state. We present Handoff-H1, a takeoff system built from three layers: purpose-built computer-vision models that extract primitives; tool-using agents equipped with image operations and in-house visual-task tools, including CV-model-backed counting, detection and plan decomposition; and a persistent, hierarchically structured project foundation, grounded in a curated construction knowledge base. We evaluate on the Construction Blueprint Takeoff Benchmark: 10 real residential blueprint sets paired with consensus-validated expert takeoffs - 2,009 verified line items, restricted for scoring to the 1,348 primary-tier materials that drive an estimate - scored per trade by an LLM judge on material coverage and quantity Precision@25% (P@.25) and combined into a weighted composite. Under identical scoring from the raw PDF, seven frontier and open-weight models span composites of 35-61, and independent professional estimators - scored against the same reconciled gold standard - post 77.6% (65.5% coverage, 87.9% P@.25). Handoff-H1, working end-to-end from the raw PDF, reaches 81.6% (86.1% coverage, 78.8% P@.25): roughly 20 points above the strongest frontier agent, and above the independent estimators by pairing near-human quantity precision with coverage they do not reach. The evaluation harness is public for the open harbor framework; the blueprint sets and ground truth are available upon request for research use.
Summary
Main Finding
Handoff-H1—a three-layer system combining purpose-built vision models, a persistent hierarchical project foundation grounded in a curated construction knowledge base, and orchestrated vision agents with independent verification—produces materially better automated material quantity takeoffs from residential blueprint PDFs than general-purpose frontier models and slightly surpasses independent professional estimators on a composite score that balances coverage and quantity precision. From raw PDFs on TAKEOFFBENCH‑V1, Handoff‑H1 achieves a composite of 81.6% (86.1% coverage; 78.8% Precision@25%), versus 35–61 composite for seven frontier/open models and 77.6% composite for independent professional estimators (65.5% coverage; 87.9% Precision@25%).
Key Points
- Architecture: three complementary layers
- Perception: specialized CV and VLMs recover typed visual primitives (rooms, walls, doors, dimension chains, callouts) and expose these models as callable tools.
- Project foundation: a persistent, hierarchical representation built once per project that synthesizes perception outputs with a curated construction knowledge base (assembly conventions, specs, defaults), partitions content by trade/assembly, and resolves consistency conflicts.
- Orchestration: vision agents that use generic image ops plus task-specific tools (e.g., framing decomposition, CV-backed counting/detection) to produce trade-scoped takeoffs; an independent verification pass audits and corrects outputs.
- TAKEOFFBENCH‑V1
- 10 real, permissioned, PII‑stripped residential blueprint sets.
- Gold standard: reconciled by multiple independent professional estimators and construction experts; 2,009 verified line items total, 1,348 primary-tier items in scope for scoring.
- Gold-standard creation enforces multi-reviewer reconciliation to produce defensible ground truth.
- Evaluation and metrics
- Judging: LLM judge (OpenAI gpt‑5.5) matches predicted to ground-truth items per trade (many‑to‑many allowed), judged 10× and averaged.
- Metrics: Coverage (fraction of ground-truth primary items with a matching prediction) and Quantity Precision@25% (P@.25: matched items with predicted quantity within ±25%). Composite score = coverage^0.4 * precision^0.6.
- Rationale: multiplicative composite penalizes missing items and inaccurate quantities; exponent weights precision more heavily to reflect greater reliability of contained ground-truth items vs omissions.
- Comparative results
- Handoff‑H1: composite 81.6% (86.1% coverage; 78.8% P@.25).
- Seven frontier/open models (same PDFs/harness, run as tool-using agents): composite range 35–61.
- Independent professional estimators (same scoring): composite 77.6% (65.5% coverage; 87.9% P@.25).
- Interpretation: Handoff‑H1 trades a few points of per-item quantity precision for substantially higher coverage versus humans, yielding a more thorough takeoff overall.
- Dataset / access model
- Harness is public; blueprint sets and reconciled ground truth are access-controlled (available on request) to avoid training-corpus contamination.
Data & Methods
- Dataset
- 10 residential drawing packages (floor plans, foundations, elevations, sections, roof plans), PII-stripped.
- Focus limited to primary-material line items (framing members, sheets, structural quantities). Secondary items and labor excluded to keep gold standard verifiable.
- Final reconciled gold standard: 2,009 total line items, 1,348 primary-tier in scoring scope.
- Ground-truth process
- Multiple independent professional estimators (≥5 years’ experience) produced takeoffs to a fixed instruction set.
- Internal construction experts audited submissions for structure and then for coverage/precision, performing hand corrections.
- A takeoff accepted by ≥2 experts and ≥2 independent estimators becomes the inter‑estimator‑approved gold standard.
- Perception and agent tooling
- Purpose-built CV/VLMs trained/finetuned for construction drawing primitives (room/function classification, wall/wall-run extraction, symbol counting, dimension strings, scales).
- Vision agents use a two-tier tool set: generic image ops (crop/zoom/pan) and task-specific tools (framing decomposition, CV-backed counting/detection). Agents choose when to delegate to specialized tools.
- Project foundation construction
- Extraction: dense semantic retrieval over perception outputs, with provenance and authority tags.
- Synthesis: partitioning by construction clusters/trades and grounding each partition against the curated KB (assembly conventions, spec defaults).
- Consistency resolution: algorithmic or agentic conflict resolution within partitions.
- Foundation persisted across estimation runs.
- Scoring mechanics
- LLM judge (gpt‑5.5) performs item matching and evaluates quantity closeness.
- Each prediction set judged 10 times to average stochasticity.
- Composite formula: score = coverage^0.4 * precision^0.6 (Precision = P@.25).
- Baselines and ablations
- Compared against seven frontier/open models run as agents on same PDF inputs under the same harness.
- Human baseline derived from independent estimators scored against reconciled gold standard.
Implications for AI Economics
- Labor substitution and augmentation
- Handoff‑H1 demonstrates that domain-specialized AI can reach or exceed human-level composite accuracy on routine quantity-takeoff tasks, indicating potential for substantial automation of mid-skilled estimation work.
- Near-human precision + higher coverage implies fewer missed items (reducing underbids/overruns) and less rework—productivity gains for firms that adopt such systems.
- Demand shift: decreased need for manual line-item counting; increased demand for higher‑value estimator tasks (complex assemblies, judgment calls, exceptions, contract negotiation, quality control, verification of edge cases).
- Value of domain-specific assets
- The system’s performance depends heavily on a curated construction knowledge base and purpose-built vision models. Firms that build and maintain proprietary KBs and annotated CV assets have a competitive advantage, raising barriers to entry and concentrating gains among adopters/investors.
- Persistent project foundations (reusable, incremental knowledge) create economies of scale across projects and justify ongoing curation investment.
- Pricing, procurement, and risk
- Small percentage errors in quantity scale to large dollar impacts on construction projects; automated systems must balance coverage vs per-item precision. Handoff‑H1’s higher coverage may reduce the common human failure mode of missed items, but the slightly lower P@.25 vs humans highlights remaining risk for cost-sensitive line items.
- Procurement and contracting may adapt: buyers may demand AI-verified takeoffs or shift risk allocations, and insurers/clients may require audit trails and independent verification for AI-generated estimates.
- Market structure & benchmarks
- Benchmarks that withhold gold data (access‑controlled) prevent training contamination, preserving long-term validity. This access model favors organizations that can obtain or license datasets, again favoring incumbents with domain ties.
- Public evaluation harnesses plus controlled data access are a pragmatic compromise for measuring progress while avoiding benchmark leakage.
- Complementarities & organizational change
- Adoption will likely be hybrid: AI handles routine extraction/measurement and flags exceptions; human experts focus on synthesis, judgment, and client communication. Organizational roles will tilt toward AI‑supervision, KB curation, and exception handling.
- Investment in data capture, annotation, and internal KB curation becomes a strategic capability for construction firms and software vendors.
- Policy and adoption frictions
- Verification and liability frameworks will be needed (who bears cost when AI-driven quantities are wrong?). Standards bodies or industry associations may develop certification/validation practices for automated takeoff tools.
- Transition effects: short-term displacement risk for junior estimators, but potential net gains in throughput and margin capture for firms that integrate AI effectively.
- Open questions for further economic study
- Generalizability beyond residential plans and to non-primary items (secondary materials, labor) remains untested; these are economically significant and harder to standardize.
- Long-run impact on estimator wages, training investments, and how task reallocation affects firm-level productivity and market concentration.
- How data governance (access-restricted benchmarks, proprietary KBs) shapes diffusion of capability and competitive dynamics across firms and vendors.
Limitations to note for economic interpretation - The benchmark focuses on primary-material residential takeoffs and omits labor/secondary items, so estimated economic impact is bounded by that scope. - The curated KB and tooling are proprietary; reproducibility and broader generalization depend on access to similar assets. - Some per-item precision remains below human experts; for high-dollar or heavily penalized items the human-in-the-loop or verification steps will still be required.
If you want, I can: - Extract the core numeric results and a short slide-ready summary. - Map estimated labor/time savings and a simple back-of-envelope economic model (cost savings per project) using typical estimator rates and project sizes.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents a clear, reproducible-sounding evaluation on a purpose-built benchmark with reconciled human ground truth and compares the system to multiple baselines and independent estimators; however the dataset is small (10 projects), model and KB details are proprietary/withheld, evaluation data are gated (available on request), and potential overfitting and benchmark-selection biases are not ruled out.
Methods Rigormedium — The authors describe a careful ground-truth reconciliation process, per-trade metrics, and stochastic LLM judging repeated across runs, and they evaluate against both frontier models and human estimators; but important experimental details are omitted (specific model identities, prompts, tooling/orchestration specifics, ablations), the sample is limited to 10 residential projects and primary materials only, and no statistical tests or robustness analyses (e.g., holdout sets, cross-validation, sensitivity to KB changes) are reported.
SampleTAKEOFFBENCH-V1: 10 real, permissioned residential blueprint sets (PII-stripped) comprising full drawing packages (floor plans, foundation, elevations, sections, roof plans). The reconciled ground truth contains 2,009 verified line items, of which 1,348 are primary-tier materials in scope for scoring. Nine trades are covered. Independent professional estimators (>=5 years experience) produced initial takeoffs that were audited and reconciled by internal construction experts; evaluation uses an LLM judge (OpenAI gpt-5.5) run 10x per prediction set.
Themesproductivity human_ai_collab adoption
GeneralizabilitySmall, non-random sample: only 10 residential projects — may not represent broader construction heterogeneity., Residential-only: results may not generalize to commercial, industrial, infrastructure, or international building codes/conventions., Primary-materials only: excludes secondary items and labor, so economic impact on full estimating workflows is unclear., Proprietary curated knowledge base: domain logic and heuristics are not disclosed and may encode organization-specific practices., Withheld model and orchestration details: limits reproducibility and external verification., Benchmark gating and potential training-contamination risks: data available upon request, raising the possibility of subtle dataset leakage into systems under test., LLM-based judge introduces stochasticity and potential matching biases not equivalent to human adjudication.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| TAKEOFFBENCH-V1 contains 10 real, permissioned, PII-stripped residential blueprint sets with 2,009 verified line items, of which 1,348 primary-tier material items are included in scoring. Output Quality | null_result | Benchmark size and number of scored primary-material line items |
Reading fidelity
high
Study strength
medium
|
n=10
2,009 verified line items; 1,348 primary-tier items in scope
|
| Handoff-H1 achieved an 81.6% precision-weighted composite score on TAKEOFFBENCH-V1, consisting of 86.1% material coverage and 78.8% quantity Precision@25%. Output Quality | positive | Completeness and quantity accuracy of primary-material takeoffs |
Reading fidelity
high
Study strength
medium
|
n=10
81.6% composite (86.1% coverage, 78.8% P@.25)
|
| Handoff-H1's composite score was roughly 20 percentage points higher than the strongest frontier or open-weight model evaluated under the same raw-PDF scoring setup. Output Quality | positive | Composite takeoff performance relative to general-purpose frontier and open-weight models |
Reading fidelity
high
Study strength
medium
|
n=7
roughly 20 points above the strongest frontier agent
|
| Independent professional estimators achieved 65.5% material coverage, 87.9% quantity Precision@25%, and a 77.6% composite score on the same reconciled gold standard. Output Quality | positive | Human takeoff coverage, quantity precision, and composite performance |
Reading fidelity
high
Study strength
medium
|
77.6% composite (65.5% coverage, 87.9% P@.25)
|
| Compared with independent professional estimators, Handoff-H1 produced substantially higher material coverage but lower quantity precision: 86.1% versus 65.5% coverage and 78.8% versus 87.9% Precision@25%. Output Quality | mixed | Tradeoff between takeoff completeness and quantity accuracy |
Reading fidelity
high
Study strength
medium
|
n=10
86.1% vs. 65.5% coverage; 78.8% vs. 87.9% P@.25
|
| The benchmark composite is calculated as coverage^0.4 multiplied by quantity Precision@25%^0.6, giving greater weight to precision than coverage. Output Quality | null_result | Composite evaluation metric for material takeoff quality |
Reading fidelity
high
Study strength
high
|
score = coverage^0.4 · precision^0.6
|
| The reported benchmark metrics are averages over 10 stochastic LLM-judge runs for each prediction set rather than results from a single judge sample. Output Quality | null_result | Evaluation-measurement procedure and score aggregation |
Reading fidelity
high
Study strength
high
|
n=10
10 judge runs per prediction set
|
| Handoff-H1 is organized into three layers: a perception layer for typed visual structure, a knowledge layer containing the persistent project foundation and curated construction knowledge base, and an orchestration layer for scoped estimation and verification. Other | null_result | System architecture and functional decomposition |
Reading fidelity
high
Study strength
low
|
not reported
|