Benchmarks that measure only correctness understate real-world risk: many AI errors are hard and costly for humans to detect, so evaluation should measure the verification effort required under realistic user budgets.

Main Finding

Current AI evaluation that reports correctness (benchmark accuracy) is insufficient. Reliability in deployment is determined not just by whether outputs are correct, but by how costly it is for users to detect incorrect outputs. The paper introduces Verification‑Cost Errors (VCEs): incorrect input–output pairs that a declared fraction of the verifier population fail to detect within the verification budget available in a deployment. Evaluation should therefore measure verification cost relative to deployment budgets, not correctness alone.

Key Points

• Verification gap: Generating outputs (Cg) is often cheap; verifying them (Cv) can be orders of magnitude more expensive (Cv ≫ Cg). This asymmetry enables errors that are hard to detect in practice.
• VCE definition: Operational — an incorrect output becomes a VCE when a specified fraction of verifiers cannot identify the error within the available verification budget.
• Failure class emphasized: outputs that are stylistically fluent, locally coherent, and authority‑mimicking are particularly likely to produce VCEs (deceptive plausibility, not necessarily identified by traditional “hallucination” labels).
• Limits of current framings:
  • Hallucination/factuality: Identifies incorrectness but not detectability under resource constraints.
  • Calibration/uncertainty: Helps triage but does not reduce verification cost and can miss systematic, high‑confidence errors.
  • Benchmarks: Treat all errors as uniform, hiding heterogeneity in detection effort.
  • Explanations and tool‑augmentation: May increase perceived trust or shift verification burden (e.g., verifying retrieved evidence or tool outputs), sometimes worsening human oversight cost.
• Empirical motifs and examples:
  • Code generation: passes supplied tests but fails in edge cases; detecting requires extra testing or manual inspection.
  • OCR + LLM pipelines: character errors get smoothed into plausible but semantically wrong text, requiring source comparison.
  • RAG: grounding can create a veneer of authority; users must verify claim–source alignment.
  • Cited empirical findings (as motivating evidence): developer studies showing increased task time with AI assistance, RAG tools still hallucinating at nontrivial rates.
• Proposal: verification‑aware benchmarking and explicit measures (authors propose constructs such as success‑probability and observed‑burden; treat verification cost as an axis of evaluation). The paper frames this as a conceptual instrument rather than a finished metric.

Data & Methods

• Conceptual/formal constructs:
  • Define generation cost Cg(x,y) and verification cost Cv(x,y) for input x and output y.
  • Formalize VCEs as incorrect (x,y) pairs that fail detection by a declared fraction of a verifier population within a specified verification budget.
  • Introduce related measures (not fully standardized in the excerpt): success‑probability (probability a verifier detects the error within budget) and observed‑burden (empirical resources consumed to verify).
• Empirical support (illustrative, not a large new dataset):
  • Case evidence from code generation, multi‑modal document understanding, and literature citations (e.g., Becker et al. 2025, Magesh et al. 2025, Dahl et al. 2024) documenting (i) hidden bugs or hallucinations that survive naive checks and (ii) increased human verification time in practice.
• Methodological stance:
  • The paper is primarily conceptual and diagnostic; it proposes a benchmarking methodology in later sections (Section 6) to operationalize verification cost, but emphasizes the construct over a finalized metric.
• Limitations noted by authors:
  • VCE measurement depends on the chosen verifier population and budget; creating standard, comparable protocols is nontrivial.
  • The work does not claim empirical prevalence numbers for VCEs across all domains; it identifies the measurement gap and proposes a way forward.

Implications for AI Economics

• Hidden labor costs: Deploying generative AI shifts verification effort onto users or downstream workers. Economists and firms must account for verification labor as part of total cost of ownership (TCO) for AI systems.
• Productivity accounting: Claims of productivity gains from AI may be overstated if they ignore increased verification time. Measured output per worker could decline once verification overheads are included.
• Market for verification services and tools:
  • Demand for verification specialists, tooling, testing suites, provenance/audit systems, and certification services is likely to grow.
  • New business opportunities (and costs) arise for third‑party auditors and specialized verification platforms.
• Pricing and procurement:
  • Procurement decisions should incorporate verification‑cost‑adjusted performance metrics (e.g., effective correct outcomes per verification hour).
  • Vendors may need to disclose expected verification costs or VCE rates for given deployment budgets; markets may evolve to price models accordingly.
• Incentives and investment:
  • Model developers may optimize for benchmark accuracy rather than minimizing verification cost; incentive misalignment could persist unless benchmarks and procurement incorporate verification metrics.
  • Investors and managers evaluating ML projects should treat verification cost reduction as an explicit objective alongside accuracy and latency.
• Liability and regulation:
  • Regulatory compliance and legal risk depend on whether errors are detectable within practicable budgets. Standards or certification regimes may require verification‑aware reporting.
  • Liability allocation (who pays for verification failures) becomes economically significant; insurers may price coverage differently for systems with high VCE risk.
• Welfare and distributional effects:
  • Increased verification burdens may disproportionately affect smaller firms or under‑resourced users who lack time/expertise, creating asymmetric adoption or risk exposure.
  • If verification tasks concentrate in lower‑paid labor segments, there are distributional consequences even as model deployment proliferates.
• Measurement and macro indicators:
  • Macro productivity statistics and returns to AI adoption should incorporate verification costs to avoid biased estimates of AI’s economic impact.
  • Benchmarking agencies and standard setters (public and private) could create verification‑aware metrics to guide policy and investment.

Practical takeaways for economists and decision‑makers: - Don’t take reported accuracy as a full measure of deployable reliability; ask how much time/expertise is needed to verify outputs. - Incorporate Cv (verification cost) into cost‑benefit analyses, procurement specifications, and ROI models. - Support or demand benchmarking standards that report verification burden or VCE rates under realistic verifier populations and budgets.

Limitations and open questions relevant to economic analysis: - Standardizing verifier populations, budgets, and tasks is required for comparability but may be contentious and domain‑specific. - Empirical quantification of VCE prevalence and the elasticity of verification cost with model scaling remain open research areas with important economic consequences.