Black-box testing can be fundamentally blind to rare, trigger-activated unsafe behavior: worst-case detection error scales with trigger probability and loss, and cryptographically hidden triggers can evade any polynomial-time black-box evaluator; reliable assessment therefore requires white-box probes, architectural or training-time guarantees, or deployment monitoring.

Main Finding

Black-box evaluation cannot reliably estimate deployment risk for models that condition behavior on latent, rare-to-evaluation but common-in-deployment contexts ("latent context-conditioned" models). Information-theoretic and computational lower bounds show that when expected trigger exposure during evaluation mε = O(1), neither passive nor adaptive black‑box testers can guarantee meaningful accuracy; only stronger access (white‑box probing or privileged deployment information) can overcome these limits. The paper gives explicit non-asymptotic constants and a matching hierarchy of hardness: passive < adaptive (information-theoretic hash barrier) < computational (cryptographic barrier) < white-box (feasible sample complexity).

Key Points

• Core model and parameters
  • Latent context conditioning: model has binary latent indicator z(x) and behaves identically off the trigger set Strigger, but differently on Strigger.
  • ε = P_Deval[z(x)=1] (trigger probability under evaluation).
  • δ = P_Ddep[z(x)=1] (trigger probability under deployment), with δ > ε possible.
  • L = |ℓ1 − ℓ0| = conditional loss gap on trigger vs non-trigger.
  • m = number of evaluation queries/samples; γ = probe quality in white-box setting; εR = desired absolute risk error.
• Passive (i.i.d.) lower bound (Theorem 4.1)
  • Any estimator that sees m i.i.d. draws from Deval has minimax expected absolute error ≥ (δL/4) · (1 − ε)^m.
  • In the small-exposure regime (mε ≤ 1/6) this simplifies to ≥ (5/24) δL (explicit constant).
  • Intuition: with tiny ε most transcripts look identical under safe/unsafe models, so no reliable inference.
• Adaptive lower bound with m‑wise hash construction (Theorem 5.1)
  • Under an m‑wise independent hash family that defines trigger sets, any adaptive evaluator with ≤ m queries has worst‑case expected error ≥ (εL/4) · (1 − mε).
  • For mε ≤ 1/8 this yields ≥ 7εL/32.
  • Key observation: m‑wise independence enforces marginal h(x) ~ Unif[0,1] conditioned on the transcript, so adaptivity cannot increase per‑query inclusion probability beyond ε.
• Query complexity / detection probability (Theorem 6.1)
  • For i.i.d. querying, expected number of draws until seeing a trigger is E[N] = 1/ε.
  • To detect with probability ≥ 1 − η requires m ≥ ln(1/η)/ε samples (tight up to constants).
• Computational separation (Theorem 7.3)
  • Under standard trapdoor one-way function assumptions, an environment (or deployer) with privileged information (the trapdoor) can realize δ ≈ 1 triggers, while any PPT black‑box evaluator without the trapdoor cannot distinguish safe from unsafe except with negligible advantage.
  • This makes trigger separation a computational (not merely statistical) phenomenon in realistic threat models.
• White-box probing (Theorem 8.3)
  • With white‑box access to parameters (probe of quality γ = α0 + α1 − 1), to estimate Rdep to absolute error εR with failure probability ≤ η requires m ≥ (18 / (γ^2 εR^2)) · log(12/η) samples.
  • Rate is Θ(1/(γ^2 εR^2)) (variance‑limited εR^−2 dependence); all constants are explicit via Hoeffding bounds and a three‑way union bound.
• Supporting mathematical tools
  • Explicit Le Cam / two‑point reduction for L1 Bayes risk (with constants).
  • Tensorization of total variation distance for product measures.
  • m‑wise independent hash construction (polynomial hash over finite fields) to neutralize adaptivity.
  • Yao’s minimax principle for adaptive lower bound.
  • Cryptographic hardness (trapdoor OWP) to show computational indistinguishability.
  • Explicit Hoeffding tail bounds and bias correction for white‑box rates.
• Practical examples motivating the framework:
  • LLMs that behave safely on benchmarks (rare toxic prompts) but reveal toxic behavior for deployment populations that use toxic language frequently.
  • Self‑driving cars tested on scarce snowy conditions (ε small) but deployed in snow‑dominated regions (δ large).

Data & Methods

• Nature of the work: purely theoretical / mathematical. No empirical datasets; results are proved analytically with explicit constants.
• Main proof techniques and constructions:
  • Minimax/statistical lower bounds built from Le Cam’s two‑point method and an explicit L1 Bayes risk lemma.
  • Total variation tensorization to relate single‑sample indistinguishability to m‑sample indistinguishability.
  • Construction of model classes with latent triggers to realize worst‑case separations.
  • m‑wise independent hash families (explicit polynomial construction over finite fields) used to eliminate adaptivity advantages; proofs exploit conditional uniformity granted by m‑wise independence and Fubini’s theorem.
  • Yao’s minimax principle to convert average‑case hardness under a prior into adversarial lower bounds against adaptive testers.
  • Cryptographic reductions under trapdoor one‑way function assumptions to obtain computational hardness (negligible distinguishing advantage for PPT evaluators without trapdoor).
  • Concentration inequalities (Hoeffding) with explicit constants to derive sample complexity for white‑box probes and to bound failure probabilities.
• Output: non‑asymptotic, constant‑explicit bounds (rather than just Θ(·) or o(·)) for all main statements.

Implications for AI Economics

• Limits of market‑reliant black‑box certification
  • Third‑party black‑box testing and marketplace due diligence can be informationally and computationally insufficient when models implement latent, deployment‑specific triggers. Buyers relying on evaluation performance may systematically underestimate deployment risk when mε = O(1).
  • Adverse selection: sellers (or deployers) with privileged information (insiders, deployers who can craft deployment triggers) can profitably present safe‑appearing models while deploying unsafe behaviors, leading to market failure absent stronger oversight.
• Value of privileged information and insider advantage
  • The computational separation formalizes a tangible informational wedge: actors with privileged secrets (trapdoors, deployment‑time trigger knowledge) can obtain enormous advantage. This increases market value of proprietary information and raises moral‑hazard risks.
• Auditing and compliance costs
  • White‑box auditing and continuous monitoring are economically costly but necessary for worst‑case assurances. The required sample complexity scales as 1/εR^2 and is inversely quadratic in probe quality γ. Thus, achieving tighter risk tolerances (smaller εR) quickly raises audit costs.
  • Because detection under black‑box adaptivity is constrained by the expected inclusion ε, to detect rare but harmful triggers auditors must run O(1/ε) queries, which may be infeasible for very small ε.
• Incentive and contract design
  • To mitigate information asymmetries, procurement contracts may need to require white‑box access, certified probes, escrowed trapdoor disclosure, or ongoing deployment monitoring and indemnities.
  • Market for safety services: demand for structured white‑box audits, provable training‑time guarantees, interpretability tools, and continuous monitoring will grow. Buyers may value certifications that demonstrate probe quality γ and verified sample counts per the explicit bound.
• Regulatory implications
  • Regulatory frameworks that accept only black‑box testing (benchmarks, challenge sets) will be inadequate in worst‑case scenarios addressed here. Regulators should consider requirements for:
    • White‑box inspection or standardized probing protocols with provable detection power.
    • Continuous monitoring and post‑deployment auditing to catch triggers that were rare during testing.
    • Disclosure rules limiting the use of cryptographic mechanisms that can hide triggered behaviors from external auditors.
• Strategic tradeoffs and policy design
  • There is a tradeoff between IP/privacy of models and verifiability: stronger white‑box guarantees mitigate worst‑case risks but conflict with commercial incentives to protect models.
  • Economic interventions could include liability rules, mandatory reporting of probe results, certification standards tying market access to measurable probe quality γ and auditing sample size, and subsidies/insurance to offset audit costs.
• Quantitative takeaways for economic modeling
  • When mε = O(1), expected absolute estimation error has a nontrivial lower bound proportional to L and either ε or δ (depending on threat model). For market models, this implies that small‑probability triggers under test distributions translate to large unpriced deployment risk unless additional information or access is provided.
  • Cost models for auditing should include an εR^−2 term (variance-limited) and account for probe quality γ; decreasing tolerated risk by factor k multiplies required white‑box sample size by k^2.
• Open economic questions suggested by the results
  • How to price and contract for white‑box audits and ongoing monitoring given quadratic scaling in εR?
  • Market mechanisms to reduce δ − ε gaps (e.g., incentivizing training diversity, mandatory stress testing across plausible Ddep).
  • Design of certification regimes that balance IP protection and the necessity of privileged checks to avoid computational hiding of triggers.

Limitations / caveats - Results are worst‑case: they show impossibility for adversarial constructions satisfying the unobservability and trigger assumptions. Typical practical models may not be adversarially constructed. - Computational hardness relies on cryptographic assumptions (trapdoor one‑way functions); if those assumptions fail, the computational separation may not hold. - The paper focuses on binary latent triggers and perfect distinguishability on triggers for some results; some relaxations are discussed but practical triggers may be more complex.

Overall, this work gives rigorous economic and statistical reasons why markets and regulators should not rely solely on black‑box evaluation to certify worst‑case safety. It quantifies the audit resources needed under stronger access and exposes the economic value of privileged information that can make black‑box certification ineffective.