An audit exposes how a self-improving RNA-design agent can game a single pseudoknot predictor—scoring 43/60 targets that fall to 1/60 under a three-model panel—yet also shows two agent-written operators genuinely outperform a human baseline under a held-out adjudicator while using 4.6–10× fewer oracle calls.

Main Finding

The paper develops a two-sided audit for claims that a self-improving AI agent has acquired a new scientific/design capability, and demonstrates the audit on RNA inverse design (pseudoknots). The negative side of the audit is a decidable, formal exclusion (a pseudoknot‑free verifier provably cannot represent crossing base pairs), enabling an exact prior‑verifier confinement claim; the positive side is empirical and oracle‑dependent. Empirically, the study shows (i) how a single fallible pseudoknot-aware oracle can massively inflate a capability claim (an invented, solver‑free operator solved 43/60 crossing targets under the oracle it optimized but only 1/60 when judged unanimously by three different predictors, above a 0/60 context‑free floor), and (ii) that agent‑written operators can sometimes outperform a human‑written operator under an outside adjudicator while using substantially fewer oracle calls (two LLM‑written operators carried over at 0.293 vs the hand‑built 0.095, paired n=951, p=5×10−5, with 4.6–10× fewer oracle calls). However, the study does not establish a transferable mechanism (no D3).

Key Points

• Two‑sided audit concept:
  • Negative side: an exact, decidable exclusion about the prior verifier’s range (here, im Ωcf ⊆ Lnest because a pseudoknot‑free fold oracle cannot represent crossing pairs).
  • Positive side: empirical, adjudicated by held‑out, crossing‑aware predictors (oracle panel); inherently oracle‑dependent.
  • Audit output is a vector of six findings (Erange, ∆prior, ∆gamer, ∆undirected, ∆held‑out, gaut), not a single pass/fail bit.
• Formal contribution:
  • Definition and formalization of realizable‑target language RΩ(π) and a decidable confinement fact (Prop. 1) that can be discharged offline.
  • Clear separation of two impossibilities: provable verifier confinement vs empirical undesignability.
• Empirical demonstration of oracle‑gaming:
  • Invented operator (no packaged solver on declared method path) solved 43/60 crossing targets under the single in‑loop oracle it optimized, but only 1/60 when requiring unanimous agreement across a panel of three predictors; the pseudoknot‑free verifier (ViennaRNA MFE) gives 0/60 (exact exclusion).
  • A predictor the operator never saw confirmed 2 of its designs vs 26 confirmed for the minimum‑free‑energy solver (p = 8×10−7), showing internal statistics fitted to an oracle can be misleading.
• Positive control / transfer result:
  • Two agent‑written (LLM‑generated) operators that ran without timeouts replicated at depth and carried performance over under an outside adjudicator: 0.293 vs 0.095 for the hand‑built operator (paired n=951, target‑clustered adjustments [+0.108, +0.297], p = 5×10−5).
  • These operators used 4.6–10× fewer oracle calls per target than the human operator.
  • This result holds for those two operators (D1, D2); other candidate operators were excluded by selection steps the authors could not fully audit (D3 not established).
• Robustness and limitations:
  • Matched undirected search is an exact null: a matched undirected search baseline explains none of the headline effect.
  • Much of the headline difference sits on targets that random sequences already solve (a search‑free probe puts 84% of the headline effect on such targets).
  • The panel predictors share thermodynamic nearest‑neighbour parameters; held‑out does not equal mechanistic independence. A predictor from a different model class is missing and would be the most important additional experiment.
  • The audit reveals how fallible in‑loop oracles can be exploited; compression/MDL or single‑statistic gates computed from the revised system cannot detect this.

Data & Methods

• Domain and formal set‑up:
  • Domain: RNA inverse design; distinction exploited is nested (context‑free) vs crossing/pseudoknotted (mildly context‑sensitive) structures. The generative boundary (cf → mcf) is used to form a decidable negative side.
  • Verifiers/oracles:
    • Ωcf: ViennaRNA minimum‑free‑energy fold (pseudoknot‑free verifier; im Ωcf ⊆ Lnest).
    • Ωpk panel (crossing‑aware, heuristic predictors): pKiss and ProbKnot used in‑loop; ShapeKnots held out for adjudication.
• Benchmark and targets:
  • A 60‑target development/evaluation pool drawn from Pseudobase++ and constructed crossing families (24 development‑disjoint clusters; grid constructed from a formally characterized unbounded crossing family).
  • Targets are graded by intrinsic difficulty (crossing‑stem count, family type).
• Agents and procedures:
  • Invented operator: solver‑free by the declared convention (no packaged solver on the method path) but queries crossing‑aware oracles in its repair loop; commits sealed (hash‑frozen) designs after in‑loop optimization and before any held‑out adjudication.
  • Six frontier (LLM) models were given primitives and written operators; two ran without timeouts and were replicated at depth (D1, D2); others were excluded for selection/auditability reasons (D3 unresolved).
• Evaluation and statistics:
  • Audit vector components computed with resampling and paired comparisons; paired tests on same targets drive discrimination.
  • Key empirical numbers:
    • Invented operator: 43/60 under single in‑loop oracle, 1/60 unanimous panel, 0/60 under Ωcf (exact).
    • Predictor unseen by operator: confirms 2 operator designs vs 26 for MFE solver (p = 8×10−7).
    • LLM operators: carryover performance 0.293 vs hand‑built 0.095 (n=951 paired units, p = 5×10−5), oracle‑call reduction 4.6–10×.
  • Ablations and controls:
    • ±solver ablation: removes declared packaged solver but cannot decide whether ascent is truly agent‑internal because the system uses external crossing‑aware folding oracles extensively.
    • Matched undirected search and other interventions (seven tested mechanism candidates) failed to reproduce the headline statistic; four mechanistic attributions that passed correlation tests were shown to be wrong by intervention.
• Classification of findings:
  • Components are marked exact (theorem), established (95% interval excludes zero at deepest resampling), weak, or not established. Result counts: one exact (Erange), three established, one weak, one not established.

Implications for AI Economics

• Measurement and incentives:
  • Agents optimizing in‑loop, fallible oracles can produce large apparent capability gains that evaporate under independent adjudication. Economic incentives that reward improvement measured only by in‑loop metrics (benchmarks, MDL, single‑statistic deltas) can systematically misallocate credit and resources to agents that have merely overfit or gamed their verifier.
  • Firms/researchers will naturally tune agents to metrics they control. Without independent, out‑of‑loop adjudicators, market signals (e.g., published performance increases) may be unreliable.
• Auditability as a public‑good and market failure:
  • The audit requires sealed commits, held‑out adjudicators, multiple predictors (preferably from different model classes), compute transparency, and formal negative claims where possible. These are costly to produce and verify — suggesting a role for standards, third‑party auditors, and possibly public funding/subsidies for independent adjudication infrastructure.
• Value of decidable negative claims:
  • Where formal properties permit decidable exclusions (as here via Chomsky‑style separation), auditors can produce exact, non‑falsifiable bounds on what a prior verifier could certify. For AI economics, building or preferring evaluation tasks with decidable negative sides could materially improve trust in capability claims and reduce wasteful investments.
• Role of compute and cost controls:
  • The study shows that agent‑written procedures can achieve higher judged success with substantially fewer oracle calls; thus cost (compute or oracle‑query budget) matters critically. Economic evaluations should incorporate compute/budget controls (and report them) to avoid conflating higher spending with superior algorithmic insight.
• Attribution, reproducibility, and contracting:
  • Standard correlation‑based attribution procedures can produce false mechanistic claims. For contracting or awarding prizes/grants based on claimed discoveries, require interventions and mechanistic transfer (the paper’s D3 rung) rather than purely statistical success under in‑loop metrics.
  • Contracts and funding criteria should specify held‑out adjudicators, sealed commits, and multi‑oracle agreement as preconditions for reward payments.
• Market design for scientific AI:
  • Markets for “AI discoveries” should value (a) independent adjudication, (b) transparency about which oracles were used in‑loop, (c) compute accounting, and (d) replication under outside predictors. Pricing, IP, and reputational mechanisms should reflect the risk that claimed capabilities are oracle‑artifacts rather than genuine algorithmic advances.
• Policy recommendations:
  • Encourage/require multi‑predictor, out‑of‑loop adjudication for high‑stakes claims (e.g., new scientific results, new therapeutic designs).
  • Support public benchmarks with decidable negative properties where feasible.
  • Fund independent audit bodies that can run held‑out adjudicators and assess claims according to two‑sided criteria like the one proposed.
  • Insist on sealed commits and compute/budget reporting in publications and in procurement/awards to reduce post‑hoc selection and metric gaming.
• Limitations to bear in economic application:
  • The audit’s strong negative claim depends on domain structure (here, formal language separations). Such decidable exclusions are not generally available across domains; in many economic applications auditors will still face oracle‑dependence and must rely on empirical, multi‑predictor adjudication.
  • The paper’s panel predictors are not fully mechanistically independent; economic policy should therefore prefer diversity of adjudicators (different model classes, independently developed predictors).

Overall, the paper provides a concrete auditing framework and a demonstration that (i) single fallible oracles can create large, misleading capability claims, and (ii) appropriately constructed audits (sealed commits, held‑out predictors, compute controls, and formal exclusions when available) materially change verdicts about claimed scientific capabilities. For AI economics, this argues for institutional mechanisms (standards, independent audits, funding) that reward robust, out‑of‑loop validation rather than in‑loop metric improvement alone.