Access to a proprietary, curated evidence corpus — not just better prompting or tools — determines how well an AI can value drug assets: adding the Noah AI corpus raised coverage and completeness-aware decision-utility nearly fourfold compared with public scaffolds, which mainly improved calibration but left a strict factual ceiling.

Main Finding

For the drug-asset valuation task the limiting factor for an "AI Scientist" agent is the evidence substrate, not just reasoning scaffolds or prompting. Non‑proprietary skills and public structured tools (playbooks, verifiers, objectivity charter, adversarial red-team) improve calibration and discipline, but a proprietary curated commercial-intelligence database (Noah AI) sets the factual ceiling and therefore the decision-utility ceiling. With the proprietary data present the agent recovers nearly the full curated competitive record and produces high “informed” decision quality; without it the agent is severely coverage-limited even if its reasoning is good.

Key Points

• Experimental design: a controlled three‑arm ablation holding model/backbone constant (Claude Code / Opus 4.8 MAX and a default_14dim playbook) while varying (1) web-only baseline (A), (2) +skills + public APIs (B), (3) +proprietary Noah AI data (C).
• Benchmark: 13 single‑target drug assets stratified into 5 archetypes (S1 validated‑crowded, S2 validated×white‑space, S3 uncrowded but unvalidated trap, S4 deal‑sensitive, S5 biological void).
• Metrics separated into three axes:
  • Axis 1 factual coverage (R_gold against a curated proprietary gold set; long‑tail recall; deal recall).
  • Axis 2 methodology/objectivity (eight‑law objectivity rubric; key‑fact coverage).
  • Axis 3 decision quality (tier‑in‑range, false‑GO rate, blind-panel decision‑quality DQ, and informed‑DQ = DQ × R_gold).
• Headline numeric results (A / B / C):
  • Recall vs curated gold (R_gold): 0.25 / 0.38 / 0.96
  • Long‑tail recall: 0.26 / 0.30 / 0.93
  • Deal recall: 0.81 / 0.88 / 0.92
  • Objectivity (0–4): 3.16 / 3.30 / 3.60
  • Tier‑in‑range: 0.80 / 0.89 / 0.89
  • Decision‑quality (0–10, blind panel): 7.01 / 6.96 / 7.65
  • Informed decision‑quality (DQ × R_gold): 1.76 / 2.57 / 7.43
• Completeness‑aware utility: even with perfect prose (DQ = 10), a non‑proprietary stack capped B’s per‑cell informed utility at 10 × 0.383 = 3.83 — well below observed C performance. Thus improved reasoning cannot substitute for missing facts.
• Capability‑superset accounting: factual capacity was generously unioned across arms (A ⊂ B ⊂ C) to avoid undercrediting lower arms; even under this favorable accounting B only reaches ~0.38 coverage.
• Practical conclusion: skills/public tools improve calibration (tier correctness, objectivity) and avoid some systematic errors (e.g., A mis‑calibrated on S2), but proprietary, curated long‑tail evidence is required for near‑complete competitive situational awareness and high informed decision utility.

Data & Methods

• System and arms:
  • Backbone: Claude Code with Opus 4.8 MAX; identical compute/time budgets across arms.
  • A (plain): web search only, no rubric/skills.
  • B (+skills + public tools): default_14dim playbook (planning, public APIs: ClinicalTrials.gov, PubMed, OpenTargets, OpenFDA), citation verifier, objectivity charter, adversarial red‑team; proprietary data starved.
  • C (+data): same as B but with Noah AI proprietary exports queried first.
• Noah AI corpus statistics (available to C): ~5,322 programs, 16,547 trials, 476 deals; target‑scoped slices used for competition queries.
• Benchmark: 13 assets, 5 strata chosen to expose different expected failure modes (coverage vs calibration). 28–30 scored asset–seed cells per arm after exclusions.
• Gold keys: constructed from the proprietary commercial‑intelligence family used by C and audited/enriched by industry experts. Purposefully measures recovery of a curated proprietary diligence record (R_gold), not an independent ground truth for long‑term clinical payoffs.
• Scoring:
  • Auto metrics based on deterministic matchers (name/code based) and capability‑superset unioning for factual capacity.
  • Blind human judging: 3 independent judges per report, anonymized and shuffled across conditions; judges scored objectivity (eight laws: 0–4 each), key‑fact coverage, and decision‑quality (0–10). Reports were scrubbed of provenance to blind B vs. C where possible; A’s free‑form format was partially identifiable and interpreted alongside auto metrics.
  • Informed decision‑quality = blind-panel DQ × R_gold; additional utility proxies (hard ceiling 10×R_gold, pessimistic min proxy, geometric proxy) tested robustness.
• Sensitivity: paired analysis on fully paired cells (n=27) preserves the C advantage under both raw and capacity recall accounting.
• Limitations explicitly noted by authors: gold set is not an independent ground truth (it was built from the proprietary corpus); small asset sample; single backbone model; web_extra was empty in this study so R_union = R_gold.

Implications for AI Economics

• Value of proprietary data assets
  • Proprietary curated evidence yields large marginal returns in decision‑utility for high‑stakes knowledge work. Firms owning such datasets can convert model-level competence into substantially higher actionable value.
  • Investment in curation and long‑tail ingestion (regional, preclinical, unregistered sources) likely yields outsized economic returns relative to equivalent investment in reasoning scaffolds alone.
• Market structure and product strategy
  • Bundling models with proprietary datasets is a defensible competitive strategy: reasoning advances are necessary but not sufficient for product differentiation in knowledge‑intensive domains.
  • Licensing/custody of curated commercial intelligence becomes a durable moat — difficulty of replicating long‑tail coverage implies persistent advantage and potential for premium pricing.
• Evaluation and benchmarking practices
  • Standard LLM/agent evaluations that focus on model/prompt/playbook improvements without explicitly controlling data access risk conflating data and reasoning effects. Ablation by data access (as done here) should be routine for domain applications.
  • Benchmarks for scientific/commercial tasks should incorporate completeness-aware metrics (informed‑DQ or equivalent) rather than only prose‑level quality measures.
• Policy and welfare considerations
  • Concentration of curated domain evidence in a few proprietary datasets can create asymmetries in decision quality across organizations, affecting competition in biotech/healthcare markets.
  • Public‑interest justification for funding open, high‑quality curated corpora (or standards for data access in regulated decisions) becomes stronger when downstream decision utility materially depends on coverage.
• Productization & cost tradeoffs
  • Even if cost per token/model call favors public APIs, total utility depends on coverage; firms should weigh licensing curation costs against the marginal loss in informed decision utility and associated economic risk.
  • Automation of due diligence and valuation is feasible, but realistically requires paying for curated evidence to reach near‑human (complete) performance rather than relying on better prompting or scaffolding alone.
• Research directions relevant to economic agents
  • Quantify cost‑benefit: measure ROI of proprietary curation vs. gains from improved models or playbooks.
  • Study generality: replicate across other high‑stakes domains (e.g., M&A, litigation, materials) to map where data vs reasoning dominates.
  • Policy research: analyze market power implications and potential interventions (data sharing norms, public datasets) that affect welfare.

Caveats and limits to generalization: the “gold” here is derived from the proprietary corpus and expert curation, so the experiment specifically measures recovery of that commercial diligence record. Results strongly support the claim that curated proprietary evidence materially limits achievable decision utility in this task, but external validation with independent ground truth (long‑term outcomes) and larger, more diverse benchmarks would strengthen external generalizability.