Independent audit finds roughly one in three AI-generated clinical notes contains at least one verified error, concentrated in allergies, medications and invented details; measured failure rates swing widely (≈25% to nearly all notes) depending on the reviewer standard and verification instrument.

Main Finding

A reproducible, adversarially verified audit of three commercial ambient AI scribes on the same 142 consultations (565 notes) finds that about one note in three contains at least one verified failure: 31.3% [27.0, 35.6]. The measured failure rate is highly sensitive to the audit instrument: reviewer instruction and choice of model family change verified rates dramatically (examples below). The authors publish all 618 verified findings, evidence quotes, prompts, model versions and the full pipeline so the census can be rerun under the same standard.

Key Points

• Corpus and scope

  • 142 consultations → 565 notes across three commercial scribes (Scribes A–C).
  • Sources: 57 recorded UK primary-care (PriMock57), 45 US ambulatory (ACI-Bench), 40 authored scenarios (30 trap-seeded, 10 trap-blind).
  • Notes were generated without supplying patient records, demographics or encounter dates.

• Discovery → Verification pipeline

  • 12 discovery passes (11 targeted + 1 open) produced 13,678 candidate errors; 5,898 cleared an importance filter.
  • Adversarial verification panel: two LLMs from different families (anthropic/claude-opus-5 = “harsher”, openai/gpt-5.5 = “gentler”) instructed to refute any candidate they could defensibly refute.
  • A third model (openai/gpt-5.4) served as a tiebreaker. A finding counted if both skeptics kept it or if tiebreak upheld it.
  • 618 findings survived verification (≈10.5% of candidates). Grouping suggests ≈265 distinct underlying errors (deduplication factor ~1.2–2.3).

• Headline rates and error mix

  • Notes with ≥1 verified failure: 31.3% [27.0, 35.6].
  • If excluding failure classes a patient record would likely prefill (invented identities, invented dates), pooled rate = 24.8% [20.8, 29.0].
  • Common failure modes: wrong or missing allergy/medication information, invented patient identity/dates, and rewriting history as a physical exam on telephone consultations.
  • Authors identify a three-tier taxonomy aligned with published scribe-error taxonomies, plus one additional failure mode: recording a treatment as delivered after the clinician explicitly retracts it.

• Human adjudication checks

  • Physician author blind-adjudicated 30 items (21 verified, 9 refused); upheld 20/21 verified findings → 95.2% precision [77.3, 99.2]. All 9 panel-refused candidates in that set were judged genuine failures by the physician (i.e., the panel had excluded them under its strict standard).
  • Independent clinician (not an author) adjudicated 16 items (12 verified, 4 refused); upheld all 12 verified → 100% [75.8, 100] and judged all 4 refused candidates genuine.
  • These results indicate the panel’s strict standard excludes some genuine errors (i.e., instrument decisions matter).

• Instrument sensitivity (measured on a re-reviewed subset)

  • On a stratified fifth of candidates re-reviewed with everything else unchanged but a lenient reviewer instruction, the strict instruction verified 9.3% of those candidates while the lenient instruction verified 79.0%.
  • Under the same strict instruction, the gentler model family flagged 54.8% of sampled notes vs the harsher family’s 27.8% — roughly double. Adding the gentler family as a second opinion altered the headline by ≈1 percentage point.
  • Depending on the audit standard applied, between 28% and 97% of sampled notes can be considered to carry a verified failure.

• Comparison to prior audits

  • Published audits report widely differing error mixes (omission dominant at 54–86% of errors). This paper shows instrument differences of the measured magnitude can produce those disagreements; when counting notes as this paper does, omission shares are similar to some prior work (e.g., one study reports 18% omission over notes vs this paper’s 15.4%).

• Transparency and reproducibility

  • All 618 verified findings, transcript-side evidence, every prompt and model version, and the re-runnable pipeline are released so other researchers or buyers can repeat the census under the same published standard.

Data & Methods

• Corpus: 142 consultations; notes: 282 (Scribe A), 141 (Scribe B), 142 (Scribe C), total 565 notes.
• Discovery stage: 12 passes (11 targeted hunts for known failure modes + 1 open pass), run on anthropic/claude-opus-5; produced 13,678 candidate error annotations; 5,898 passed importance filter.
• Verification stage: adversarial "skeptics" (claude-opus-5 and gpt-5.5), each shown full note + full transcript and instructed to refute candidates whenever defensible. A third model (gpt-5.4 with high reasoning effort) served as tiebreaker. Deterministic seeds used for reproducibility; exact settings and manifests released.
• Outcome rules: a candidate counts as a verified finding if both skeptics keep it or if the tiebreaker upholds it; unparseable/failed responses count as refutations.
• Human validation: two blinded human clinicians independently adjudicated disjoint random samples of verified findings + some high-importance refused candidates to assess instrument precision and the panel’s refusal decisions.
• Grouping/deduplication: findings are not merged across passes; model-assisted grouping and hand checks estimate 265 distinct errors among 618 findings.
• Release: full dataset of findings, evidence quotes, prompts, model versions, and runnable pipeline.

Implications for AI Economics

• Measurement is a market and regulatory input
  • Reported error rates for AI scribes are not intrinsic product properties but joint outcomes of product behavior and the audit instrument (discovery heuristics, reviewer instruction, model family). Procurement, regulation, and reimbursement decisions that use headline error rates need to require standardized, transparent instruments or run reproducible third-party audits.
• Competitive signaling and perverse incentives
  • Vendors may benefit from publishing favorable audit instruments or calibrating systems to perform well on specific detectors. Absent transparent, shared standards, vendors can game audits (optimize for detector sensitivity, not clinical utility), complicating buyer comparisons.
• Liability, pricing and insurance
  • Variation in measured failure rates changes expected downstream liability and thus insurance pricing and legal risk assessments. Purchasers and payers should factor audit-instrument uncertainty into contract terms, warranty clauses, indemnities and due-diligence processes.
• Product design and integration economics
  • Many errors (invented identities/dates, some omissions) arise or would be mitigated by integrating patient records and structured context. That creates economic value for tighter EHR integration and potentially lock-in: integrated solutions can reduce certain failure classes and therefore reduce oversight costs.
• Cost of oversight and clinical labor
  • The census underscores that clinician sign-off matters (clinicians authoritatively control notes at point-of-signature). But verifying and correcting AI-suggested notes imposes monitoring costs; economics of adoption depend on how often clinicians must intervene and the time-cost per intervention. Buyers must evaluate total cost of ownership (AI + associated clinician QA) rather than vendor-reported headline error rates alone.
• Role for reproducible third-party audits as public goods
  • The authors’ open, re-runnable instrument is a model for standardized third-party assessments. Publicly available, reproducible audits reduce information asymmetries between vendors and buyers, improve market transparency, and help regulators set evidence-based standards.
• Policy and regulation
  • Regulators and standard-setting bodies should require disclosure of audit instruments and promote benchmarked evaluation protocols. Regulatory decisions (approvals, sandbox outcomes, labelling) that rest on opaque internal audits risk being misled by instrument-dependent numbers.
• Investment and product strategy implications
  • Investors and procurement officers should demand audits with transparent, reproducible instruments and take care when comparing vendor claims. Vendors that support open benchmarking and clear auditability may enjoy trust premiums; conversely, those optimizing for narrowly defined detectors may face reputational and regulatory risk.

Bottom line: error rates for deployed AI scribes are meaningful only when reported alongside the exact, reproducible audit instrument that produced them. For economic decisions—procurement, pricing, liability, and regulation—buyers and policymakers must treat the instrument as part of the measurement and demand transparency and standardized benchmarks.