An agentic LLM pipeline turns messy oncology notes into structured data at scale—averaging 0.93 F1 across 103 clinical variables—and yields 94% direct human approval in a curation workflow, sharply lowering manual annotation costs.

Main Finding

HARMON-E introduces a hierarchical, agentic LLM framework that extracts comprehensive, patient-level structured oncology data from heterogeneous EHR notes and scanned PDFs at scale. On a real-world corpus (≈400,000 clinical notes / scanned reports for 2,250 patients), the system attains an average F1 of 0.93 across 103 oncology-specific variables (100/103 > 0.85; critical fields such as biomarkers and medications > 0.95) and yields a 94.1% direct manual approval rate when integrated into curator workflows—substantially reducing human abstraction effort.

Key Points

• Scope and scale
  • Targets a comprehensive oncology data dictionary: 16 entity types and 103 attributes (biomarkers, medications, staging, surgeries, radiation, etc.).
  • Evaluated on >400,000 documents (≈180 docs/patient) covering 2,250 cancer patients and ~940,923 derived data points.
• Architectural approach
  • Hierarchical, agentic pipeline decomposing tasks into retrievers (context-aware indexing, vector + rule-based search), LLM Synthesizers (multi-step reasoning), and Collators (deduplication, dependency resolution, validation).
  • Iterative retrieval + LLM reasoning (inspired by ReAct / chain-of-thought) to reconcile multi-document contradictions, normalize temporal references, and infer implicit attributes (e.g., treatment discontinuation inferred from adverse events).
  • Scanned PDFs handled via a vision–language model for OCR and page-to-document normalization before LLM processing.
• Performance and usability
  • Mean F1 = 0.93; 100/103 variables > 0.80 F1; key clinical variables > 0.95.
  • Integration into a human curation platform produced a 94.1% direct acceptance rate by oncology abstractors, indicating high practical trustworthiness.
• Evaluation
  • Metrics include entity-level recall/precision and attribute-level accuracy; authors propose an evaluation framework aligned to clinical curation needs rather than only NER metrics.
• Novelty
  • First reported end-to-end LLM-agent application for exhaustive oncology data extraction at this scale, emphasizing patient-level synthesis across heterogeneous and longitudinal notes.

Data & Methods

• Data
  • Real-world oncology notes and scanned PDF reports: ~400,000 documents for 2,250 patients, ~940k extracted data points.
  • Heterogeneous inputs: progress notes, pathology, radiology, scanned reports.
• Ingestion & preprocessing
  • Vision–language model for scanned PDF transcription to page-level Markdown.
  • LLM calls for segmentation, document typing, and metadata extraction (dates, titles).
• Core pipeline components
  • Retrievers: context-aware indexing, vector search and rule-based filters to find candidate evidence spans across documents.
  • LLM Synthesizers: LLMs act as reasoning agents to convert retrieved evidence into structured attribute-value pairs, perform multi-step inference, and resolve contradictions.
  • Collators: merge instances across documents, enforce typing/validation, handle dependencies (e.g., medication needs diagnosis context), and output patient-level structured records.
• Pipeline strategies
  • Flexible configurations (single-step, multi-step, topical extraction, sequential document analysis) tailored to task complexity.
• Evaluation
  • Align system outputs to a gold standard; compute entity-level recall/precision and attribute-level accuracy; report F1 per variable and summary metrics.
• Key performance outcomes
  • Average F1 = 0.93; 100/103 variables > 0.85 F1; critical variables (biomarkers, medications) > 0.95 F1.
  • 94.1% of system outputs accepted directly by human curators.

Implications for AI Economics

• Labor and productivity
  • Direct substitution/complement: HARMON-E demonstrates material automation potential for high-skill, costly manual abstraction (oncology registrars/abstractors). A 94.1% direct approval rate implies major reductions in time per data point and reallocation of human reviewers to exception handling and validation.
  • Task reconfiguration: routine extraction shifts to automated pipelines; human labor focuses on edge cases, quality control, and higher-level analytic tasks (e.g., interpreting ambiguous clinical scenarios).
• Cost structure and unit economics
  • Lower marginal cost per structured data point—enables scaling RWD (real-world data) curation and could compress pricing for curated oncology datasets/services.
  • Upfront costs: system development, indexing, and validation remain nontrivial; however, high acceptance rates imply rapid amortization for large-scale data providers or payers.
• Market effects and competitive dynamics
  • Vendor consolidation risk: firms that effectively combine proprietary clinical corpora, LLM expertise, and validated agentic pipelines may capture significant share of the oncology RWD market.
  • Demand-side expansion: lower-cost, high-quality structured oncology data can stimulate downstream markets—pharma RWD studies, post-marketing surveillance, decentralized trials, and precision oncology services.
• Investment and monetization opportunities
  • SaaS/Platform offerings for automated abstraction, human-in-the-loop curation tooling, and validated oncology data feeds are attractive targets for investors and incumbents (healthcare IT, analytics vendors).
  • Ancillary products: model auditing tools, provenance tracking, and regulatory-compliance services to manage liability and trust.
• Quality, liability, and regulatory considerations
  • Need for rigorous external validation, audit trails, and human oversight to mitigate risks from model hallucinations or systemic biases in extraction.
  • Regulators and purchasers may demand transparent error rates per attribute, provenance linking to source documents, and repeatable validation protocols—creating barriers to entry but also a market for compliance tooling.
• Research & policy implications
  • Macroeconomic impact: productivity gains in clinical data curation may free resources for research but also reduce demand for certain abstraction roles—policy discussion on retraining/reskilling and job transitions warranted.
  • Data governance: broad deployment raises questions about data sharing, vendor lock-in, and standards for interoperable extraction schemas that affect market competition and data liquidity.
• Suggested economic analyses to follow
  • Quantitative cost–benefit: compare per-record curation costs before/after deployment across scales (small registries vs. national datasets).
  • Labor market modeling: estimate displacement vs. productivity-augmented demand for abstractors with scenario analyses.
  • Market sizing: project addressable market for automated oncology curation platforms and downstream RWD products under different adoption curves.
  • Risk-adjusted valuation: incorporate validation/liability costs and regulatory friction into business models.

Concluding note: HARMON-E illustrates how agentic LLM pipelines can materially change the economics of clinical data curation—driving cost reductions, enabling scale, and creating new product and regulatory needs. For stakeholders (vendors, payers, researchers, policymakers), the key next steps are rigorous independent validation, transparent error/provenance reporting, and economic models that quantify both efficiency gains and labor-market impacts.