A gaze-guided vision transformer paired with an ontology-constrained vision–language model speeds retinal reads and slashes documentation time: combined Co-Annotator guidance increased correct diagnoses per minute by 40% and reduced comment editing time by 67% without reducing diagnostic accuracy, while substantially expanding documented biomarkers.

Main Finding

Co-Annotator—an expert-distilled system combining (1) a gaze-aligned Vision Transformer (ViT) that produces fixation-aligned areas-of-interest (AOIs) and (2) an ontology-bounded vision–language model (VLM) that pre-fills editable biomarker summaries—improves trainee workflow for diagnosing wet age-related macular degeneration (wAMD) on OCT. In combined deployment across two academic centers, simultaneous AOI+VLM guidance increased correct diagnoses per minute by ~40%, reduced comment editing time by ~67%, and produced several-fold more documented biomarkers, all without reducing diagnostic accuracy.

Key Points

• Interaction design principles: guidance should be deferrable (revealed after an initial unguided pass), sparse (few precise hotspots), and evidence-anchored (each biomarker token links to image evidence).
• Two complementary guidance channels:
  • Fixation-aligned AOIs (gaze-aligned ViT) to reduce visual-search overhead.
  • Ontology-bounded biomarker drafts (fine-tuned VLM) to reduce documentation burden and constrain language to clinically valid terms.
• Expert distillation: system trained on synchronized expert gaze and dictations so guidance reflects how specialists actually look and describe OCT findings.
• Performance highlights:
  • ViT gaze-alignment and VLM fine-tuning improved diagnostic micro-AUC from 0.95 → 0.98 (expert-distilled supervision).
  • Biomarker text quality: MedBERTScore ≈ 0.867 on expert-anchored biomarker generation.
  • US2 (isolated modality evaluation, n=11 residents): AOI guidance produced a post-guidance carryover in perceptual efficiency; VLM guidance more than doubled biomarker breadth (5.8 vs 2.3 biomarkers per AMD eye) with ~83% retention.
  • US3 (combined deployment, n=8 residents, two institutions): AOI+VLM together yielded +40% correct diagnoses/min (p=0.023), −67% comment editing time (p=0.023), and large increases in biomarkers documented (several-fold; p<0.05).
• Safety and control: guidance is editable, ontology-constrained, and presented as supportive evidence rather than authoritative output.

Data & Methods

• Data collection (US1 / Expert Distillation Corpus)
  • In-house dataset: 1,155 five-line OCT scans from 231 eyes (203 patients); Zeiss Cirrus platform.
  • Expert supervision: eye-tracking on a subset (138 images) and spoken dictation on another subset (113 images). Transcriptions were mapped to a curated ontology (12 biomarker terms) after automated extraction + expert review. Augmented with OCT5k to form ~573 image–biomarker pairs for VLM training/eval.
• Gaze-aligned ViT
  • Architecture: Vision Transformer producing classification + attention rollout map on a 32×32 patch grid.
  • Training objective: binary cross-entropy classification + weighted gaze-alignment loss (cross-entropy between attention rollout and expert fixation-density maps). Hyperparameter α controls attention alignment vs classification.
  • Output: fixation-aligned AOI heatmaps usable at inference without live eye-tracking.
• Ontology-bounded VLM
  • Base: MedGemma VLM, fine-tuned with a two-stage curriculum:
  • Stage 1: domain grounding via diagnosis (binary classification) and biomarker discrimination (VQA-style yes/no).
  • Stage 2: add biomarker generation (ontology-constrained) while retaining Stage 1 tasks.
  • Constrained decoding: outputs limited to predefined biomarker ontology to reduce hallucination and ensure editability.
• User studies
  • US1: data collection from retina experts to create supervision corpus.
  • US2: controlled within-institution evaluation (n=11 ophthalmology residents) assessing each modality independently—checked safety (no drop in diagnostic accuracy) and modality-specific effects.
  • US3: combined AOI+VLM deployment across two institutions (n=8 residents) measuring throughput, documentation time, and biomarker coverage; statistical testing reported (p-values provided for key outcomes).

Implications for AI Economics

• Productivity and throughput
  • A ~40% increase in correct-diagnoses-per-minute and a ~67% reduction in documentation editing time point to substantial per-clinician throughput gains. In high-volume retina clinics, this can translate into more patients evaluated per clinician-hour or reduced backlog—improving facility utilization and potentially revenue per clinic-hour.
• Labor augmentation and task allocation
  • System targets two distinct bottlenecks (visual search and documentation) and is designed to augment, not replace, clinician judgment. This supports complementary labor models where AI increases junior clinicians’ effective diagnostic throughput and reduces supervisory burden.
  • For training programs, AOI guidance has carryover effects on perceptual efficiency—potentially lowering the time and supervision needed to train novices, reducing training cost per clinician.
• Documentation quality, billing, and downstream value
  • More complete and standardized biomarker documentation (ontology-bounded) improves auditability, enables more consistent clinical records, and may support better coding/billing capture (if billing is tied to documented findings) and quality metrics for reimbursement.
  • Structured biomarker outputs can enable downstream analytics (registry data, quality measurement) with less manual abstraction cost.
• Deployment costs and required investments
  • Upfront costs include expert data collection (eye-tracking hardware, specialist time for dictation and review), model development, and integration into clinical workflows. The authors publicly released the corpus and models, lowering replication and adoption costs.
  • Ongoing costs include validation and maintenance across scanner types, software integration with EHRs, and medicolegal risk management.
• Adoption barriers and risks
  • Generalizability: dataset focuses on Zeiss Cirrus and a limited set of experts; cross-scanner and cross-population validation are required before scale-up. Economic value depends on robust performance in diverse real-world settings.
  • Liability and regulatory hurdles: systems that influence clinical decision-making face regulatory scrutiny and may require clinical trials and device clearance—affecting time-to-market and costs.
  • Behavioral risk: potential for over-reliance or miscalibrated trust, especially if guidance is misinterpreted. The interaction design mitigates this (deferrable, editable), but training and governance are needed.
• Market and business-model considerations
  • Value proposition: sell as a clinical-decision-support (CDS) tool improving throughput and documentation—payoff could be subscription/licensing to hospitals, or per-study fees tied to volume.
  • Cost-benefit sensitivity: economic gains depend on baseline clinician speeds, billing rules, and clinic volumes. Small clinics with low volume may find ROI marginal, whereas high-throughput specialty clinics could realize rapid payback.
• Broader economic externalities
  • Data/public-good effect: public release of models and corpus can accelerate diffusion, lower entry barriers, and intensify competition—driving down integration costs but also commoditizing the technology.
  • Workforce effects: augmentation may shift physician time away from rote documentation and searching toward higher-value tasks (patient counseling, complex cases), but it may also compress staffing needs in some settings; net employment effects depend on clinic-level staffing models and regulations.

Overall, Co-Annotator demonstrates that targeted, expert-distilled multimodal AI—designed around concrete workflow bottlenecks and clinician control—can yield measurable productivity and documentation benefits. From an AI-economics perspective, the largest value stems from throughput gains, reduced transcription cost, and improved documentation quality, but realizing that value at scale requires investment in cross-site validation, integration, regulatory clearance, and user training.