A practical audit protocol for AI research agents lets auditors preregister claims, withhold run histories, and run randomized paired tests to show whether discoveries are recoverable or driven by feedback; in lab benchmarks the protocol produced zero recoveries across 96 challenger episodes and detected substantial feedback effects, proving feasibility though not yet broad generality.
Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
The paper introduces the Discovery Certification Protocol, a preregistered, executable auditing framework that certifies whether claimed AI research outcomes are non-recoverable by matched challengers and whether truthful experimental feedback causally improves agent performance, and it demonstrates feasibility on controlled optimization benchmarks.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
AI research agents combine prior knowledge, public sources, and experimental feedback to produce useful results. The Discovery Certification Protocol (DCP) turns claims about these results into executable recovery and feedback tests. Gate 1 validates useful improvement on sealed evaluation. Gate 2 gives matched agents the registered starting information and observed Web content while withholding the target research history. Every valid method reaching the numerical target supplies a recovery witness and triggers the Core veto. DCP Core requires adequate controls, zero observed recoveries, and a finite-sample bound on recovery in one fresh registered episode. Optional Gate 3 measures the average effect of truthful feedback relative to a specified neutral policy from a shared checkpoint. DCP Evidence adds this effect after independent null calibration and a registered effect margin. Two controlled audits exercise the complete protocol in SQLite optimization and virtual catalyst control under different models. Each produced zero recoveries in 96 episodes, with an upper bound of 0.0468. Each paired study yielded 30 truthful recoveries and zero neutral recoveries, with passing 60-pair null studies. Additional cases exercise Core, recovered, and audit-incomplete decisions. A deterministic, LLM-free verifier reproduces the decisions from frozen evidence. DCP provides a common evidence language for useful outcomes, alternative routes, and feedback effects across AI research.
Summary
Main Finding
High scores alone are insufficient to certify that an AI research agent actually “discovered” a useful result. The Discovery Certification Protocol (DCP) provides a practical, domain‑agnostic audit protocol that (a) ties a claimed useful outcome to an executable sealed test, (b) tests whether the outcome can be recovered by matched challengers given the registered starting information, and (c) (optionally) measures the causal contribution of interactive feedback. Applied across controlled software-optimization and virtual-experiment tasks, DCP distinguishes unrecoverable claims (certifiable) from recoverable outcomes (certificate refuted) and quantifies an upper bound on the probability that a matched agent would recover the same outcome at the same budget.
Key Points
- DCP organizes an audit around a single numeric outcome and a strict information boundary. It specifies five objects: K (background), E0 (initial observations), L (target run’s research history), A (final artifact), and P (executable validity/check).
- Three gates:
- Gate 1 (useful improvement): sealed evaluation must show a meaningful improvement over baseline (pre-registered minimum gain).
- Gate 2 (recovery test / Core): fresh matched challengers receive K and E0 and the exact Web bytes observed by the target (Wobs) but not L*. Any valid challenger artifact meeting score ≥ x − ε is a qualified recovery witness; any such witness refutes Core. If zero qualified recoveries are observed in n independent episodes, DCP computes a finite-sample upper bound p_up on the per‑episode recovery probability and certifies Core only when p_up ≤ ρ and controls are adequate.
- Gate 3 (Evidence, optional): from a shared checkpoint, run randomized paired branches that receive either truthful feedback or a neutral (non‑directional) policy. Estimate Δ_feedback = E[u(A_truthful) − u(A_neutral) | checkpoint]. Require independent null calibration (neutral vs sham) and a pre-registered effect margin. Evidence requires a lower confidence bound on Δ_feedback above the pre-registered threshold.
- Recovery witnesses are constructive: any valid alternative route to the same numeric outcome under the registered information triggers Core veto (i.e., shows the outcome was recoverable without the claimed run history).
- Decision semantics are separated: Core (certificate eligibility) is distinct from Evidence (causal feedback effect), and audits can also be “incomplete” when controls or registration conditions are not met.
- Implementation details to ensure auditability: pre-registration, sealed evaluation, a recording gateway for Web bytes, deterministic verifier that re-evaluates records (LLM‑free), positive/negative control checks, and fixed error budgets.
Data & Methods
- Formal objects and registration: K, E0, L, A, P are fixed before the target run. Registration also fixes validity rules, budgets, selection rules, stopping criteria, and statistical analysis plans.
- Recovery rule: Valid(a) AND score(a) ≥ x − ε defines a recovery. ε is pre-registered with 0 ≤ ε < δ_min (the smallest useful gain).
- Finite-sample recovery bound: Suppose we run n independent challenger episodes under the registered generation procedure and observe zero recoveries. For a pre-specified per-audit Type I risk α, the protocol uses the bound p_up = 1 − α^(1/n) as an upper bound on the per-episode recovery probability (the paper reports examples using α = 0.01). Core requires p_up ≤ ρ (a pre-registered tolerable recovery probability).
- Example reported: two main tasks had 0/96 recoveries and an upper bound p_up ≈ 0.0468 (consistent with α ≈ 0.01 and n = 96).
- Gate 3 pairing / calibration: Randomized paired runs (truthful vs neutral feedback) from a shared checkpoint c; estimand Δ_feedback conditional on c. Independent null tasks (sham) test that the neutral channel is truly non-informative within a pre-registered equivalence band ±δ_sham. Evidence requires LCB(Δ_feedback) ≥ δ_evidence + δ_sham.
- Experiments (controlled audit cases):
- SQLite-Web optimization (DeepSeek-v4-flash): task of selecting 4 partial indexes among 16 query families; sealed deterministic evaluator; main run scored 0.8855; Gate 2: 0/96 recoveries; Gate 3: paired runs showed a positive feedback effect, passed null calibration.
- Virtual catalyst optimization (DeepSeek-v4-pro): 5 controls with 8 levels each (32,768 recipes), deterministic virtual lab; main run scored 1.0 (baseline 0.5990); Gate 2: 0/96 recoveries; Gate 3: paired runs showed a large positive effect, passed null calibration.
- Device calibration (diagnostic): 0/80 recoveries with adequate controls (Core passed).
- Multidimensional knapsack (developmental case): a matched challenger produced legal solutions above the recovery line, triggering a qualified recovery witness and refuting Core.
- Reproducibility: all decisions are reproducible by a deterministic, LLM‑free verifier that replays the recorded evidence bundle. Evidence bundles include Web capture (model-visible bytes), committed artifacts, and sealed scores.
Implications for AI Economics
- Measuring agent contribution to productivity: DCP supplies a practical framework to quantify when an AI agent’s output represents a reproducibly non-recoverable improvement (i.e., likely depended on its private adaptive research history) versus when the same numerical result is recoverable given publicly available inputs. This distinction matters when assigning economic credit or estimating the marginal productivity of AI assistance in R&D.
- Valuation and contracting: certification (Core pass + Evidence) can serve as a credential for procurement, licensing, or milestone payments where buyers pay premiums for irreproducible discoveries (novelty tied to agent‑specific search/experimentation). Conversely, outcomes that are recovered easily under the registered information boundary justify lower payments.
- Incentive design and information rents: by making recoverability and feedback contribution explicit and measurable, organizations can design incentives that reward genuinely novel/irreproducible discoveries while accounting for gains achievable by routine engineering or public-knowledge recombination.
- Market structure and competition: DCP-style audits could become a component of marketplaces for automated research outputs (e.g., code, assays, experimental recipes), enabling buyers to prefer certified (non-recoverable or feedback‑driven) artifacts and thus affecting pricing and competitive dynamics.
- Policy, regulation, and standards: regulators and standards bodies can use outcome-level certification rules like DCP to set minimal auditability requirements for AI-generated scientific or engineering claims—especially in high‑stakes domains (drug discovery, materials, safety‑critical systems). The protocol’s emphasis on pre-registration, sealed tests, and independent replayable evidence aligns with reproducibility and audit standards.
- Empirical research agenda in AI economics: DCP provides a concrete experimental scaffold to study (a) how much of measured R&D acceleration from AI comes from redistribution of known components vs. genuinely irrecoverable breakthroughs, (b) the value of interactive feedback in agent workflows, and (c) how information boundaries and tooling choices shift recoverability probabilities—inputs essential for models of R&D productivity, adoption, and returns to AI capital.
- Cost/benefit and adoption thresholds: the protocol highlights that certifying discovery requires extra audit overhead (recording, pre-registration, independent runs). Firms will trade off these audit costs against increased market value or regulatory compliance. Understanding these tradeoffs informs adoption trajectories and pricing for audit services and certification.
If you want, I can: (a) extract the formal definitions and equations into a concise checklist you could use to pre-register a DCP audit for a particular domain, or (b) sketch a simple economic model that values a certified (Core+Evidence) AI discovery versus an uncertified high‑scoring outcome. Which would be most useful?
Assessment
Paper Typetheoretical
Evidence Strengthmedium — The paper proposes a rigorous, well-specified protocol and demonstrates it on multiple controlled, deterministic benchmark tasks (SQLite-Web, virtual catalyst, calibration and knapsack cases) with preregistration, randomized pairing for feedback tests, and a deterministic verifier; however, empirical demonstrations are limited to synthetic/lab environments, a small number of tasks and episodes, and specific model endpoints, so external validity and breadth remain constrained.
Methods Rigormedium — Design is strong on preregistration, sealed evaluation, explicit information boundaries, randomized paired comparisons, positive controls, and a deterministic verifier, which together address many common threats (data leakage, selective reporting, measurement ambiguity); but the empirical calibration rests on a small set of toy/bench tasks, a limited set of models/endpoints, and assumptions about independent episodes and frozen Web captures that may be harder to meet in more complex, real-world research settings.
SampleControlled demonstration audits across four cases: (1) SQLite-Web optimization with 16 query families and one partial-index candidate per family; sealed evaluator over deterministic VM work, target run scored 0.8855; Gate 2 ran 96 independent no-lineage challenger episodes. (2) Virtual catalyst optimization: 5 controls, 8 levels each (32,768 recipes), virtual deterministic sealed environment; baseline 0.5990, target 1.0; Gate 2 ran 96 challenger episodes and Gate 3 ran paired branches (n=30 pairs reported for feedback effect). (3) Device calibration (24 controls) used as calibration with 80 episodes. (4) Multidimensional knapsack developmental case produced recovered witnesses. Models/endpoints included DeepSeek-v4-flash and DeepSeek-v4-pro via Claude CLI; Web interactions were recorded and replayed via a recording gateway; all candidate artifacts were committed prior to sealed scoring.
Themesgovernance innovation
IdentificationPrecommitment and sealed evaluation to rule out data leakage and hindsight claims; Gate 2 uses matched fresh agents given the same registered background and observed Web bytes but withholding the target run's research history to test recoverability (a counterfactual containment strategy); Gate 3 uses randomized paired branches (truthful vs neutral feedback) starting from a shared checkpoint to identify the causal effect of feedback, with independent null calibration and preregistered equivalence margins.
GeneralizabilityDemonstrations confined to synthetic or simulated laboratory tasks (SQLite VM, virtual catalyst) rather than large-scale, real-world research problems., Evaluations used a small number of task types and specific model endpoints (DeepSeek variants via a particular provider), so performance and protocol behavior may differ with other models or broader agent architectures., Assumes ability to capture and freeze exact Web-visible bytes and to create truly independent fresh episodes—may be hard in open, dynamic web environments or for agents with undisclosed internal state., Finite-sample bounds reported are task- and budget-specific and do not directly translate to complex, high-dimensional scientific discovery domains., Recovery rule and thresholds require careful domain-specific registration; applicability depends on whether outcomes admit sealed deterministic evaluation.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The DCP Core decision requires useful improvement, adequate controls, zero qualified recovery witnesses, and an upper bound on fresh-episode recovery probability no greater than the registered threshold. Governance And Regulation | positive | Certification eligibility and bounded recoverability of a research outcome |
Reading fidelity
high
Study strength
high
|
not reported
|
| The SQLite-Web agent improved the sealed workload score from a baseline of 0 to 0.8855, corresponding to an 88.55% reduction in virtual-machine work relative to the no-index baseline. Output Quality | positive | Sealed SQLite workload performance |
Reading fidelity
high
Study strength
high
|
0.8855 sealed score; 88.55% reduction from the no-index baseline
|
| The virtual catalyst optimization agent improved the score from a fixed baseline of 0.5990 to 1.0, selecting all five hidden optimal control levels. Output Quality | positive | Virtual catalyst recipe performance |
Reading fidelity
high
Study strength
high
|
n=256
1.0 final score versus 0.5990 baseline
|
| Neither the SQLite-Web nor virtual-catalyst audit produced a recovery among 96 matched no-lineage episodes, yielding an upper recovery-probability bound of 0.0468. Research Productivity | null_result | Recovery of the target outcome by matched agents without the target research history |
Reading fidelity
high
Study strength
medium
|
n=96
0/96 recoveries; upper bound 0.0468
|
| The SQLite-Web and virtual-catalyst audits passed their positive controls in all 45 control episodes, implying a recovery-detection recall lower bound of 0.8889 above the registered minimum of 0.8. Error Rate | positive | Positive-control execution and recovery-detection recall |
Reading fidelity
high
Study strength
medium
|
n=45
45/45 positive controls; recall lower bound 0.8889
|
| Truthful experimental feedback produced 30 recoveries while neutral feedback produced zero recoveries in each paired primary audit, and the associated 60-pair null studies passed calibration. Output Quality | positive | Recovery under truthful versus neutral experimental feedback |
Reading fidelity
high
Study strength
medium
|
n=30
30 truthful recoveries versus 0 neutral recoveries; 60-pair null studies passed
|
| The device-calibration case produced zero recoveries in 80 episodes and an upper recovery-probability bound of 0.0477, leading to a DCP Core decision. Research Productivity | null_result | Recovery of the device-calibration target without the target research history |
Reading fidelity
high
Study strength
medium
|
n=80
0/80 recoveries; upper bound 0.0477
|
| The multidimensional-knapsack diagnostic was refuted by a qualified recovery witness: a matched episode produced legal solutions scoring 0.9363 and 0.9356, both above the registered recovery line of 0.9329. Research Productivity | negative | Alternative recovery of the target knapsack solution |
Reading fidelity
high
Study strength
high
|
n=1
0.9363 and 0.9356 versus recovery line 0.9329
|
| The deterministic, LLM-free verifier reproduced the reported protocol decisions from frozen evidence bundles. Governance And Regulation | positive | Reproducibility and consistency of audit decisions |
Reading fidelity
high
Study strength
medium
|
not reported
|