Grant LLM agents wide sandboxed freedom but limit real-world authority: Progressive Risk Vesting debits a shared risk escrow only when branches request irreversible capabilities, guaranteeing episode-level catastrophic-risk bounds under explicit conditional-certificate assumptions; in a stylized branching model, the authority reproduction rate determines whether lowering local failure probability reduces total risk linearly, by its square root, or not at all.

Main Finding

Progressive Risk Vesting (PRV) separates sandboxed exploration from irreversible authority by holding a trajectory-level risk escrow and debiting it only when a branch requests an action or capability that can cause externally observable harm. Under a mild statistical condition—each activation’s certificate bounds harm probability conditional on the full pre-activation history—this per-activation accounting yields an anytime guarantee that the episode-level catastrophe probability is bounded by the root escrow. PRV therefore permits broad, cheap sandbox search while limiting realized exposure, and it produces tractable risk/compute prices and operational rules for when to grant authority.

Key Points

• Distinction: sandbox tree (untrusted, externally harmless exploration) vs authority tree (nodes that have been granted irreversible capabilities).
• Progressive Risk Vesting (PRV): governor assigns each approved activation i a conditional allowance ai with P(H_i | F_{i-1}) ≤ ai, keeps ∑ ai ≤ δ in escrow, and atomically debits ai at activation. Rejected or cancelled branches consume compute but no risk allowance.
• Anytime harm bound (Theorem 1): if every approved activation satisfies the conditional certificate and total allowances do not exceed δ, then P(any activation causes catastrophe) ≤ δ. This holds for adaptively generated, countable trees without independence or depth limits.
• Option value (Proposition 1): delaying vesting until activation (PRV) weakly improves achievable policies relative to “spawn charging” (charging risk at spawn time) because spawn charging irreversibly ties up allowances for branches that may never be activated.
• Selection / calibration caution: certificates must be valid conditional on the information used to select branches. Marginal risk estimates calibrated ex ante can understate conditional risk after selective activation (endogenous selection bias).
• Authority-reproduction phase transition (Theorem 2): model authority-bearing nodes as a branching process with mean authority reproduction number R_A = m s (mean sandbox candidates m × probability s a candidate inherits authority). As local per-node harm p → 0:
  • If R_A < 1 (subcritical): trajectory harm R(p) ~ p/(1 − R_A) (linear in p).
  • If R_A = 1 (critical): R(p) ∼ sqrt(2p/β) (proportional to √p).
  • If R_A > 1 (supercritical): lim_{p→0} R(p) = 1 − ξ > 0 (nonzero floor). This shows nonlinear/systemic amplification of local risk depending on R_A.
• Multitype occupancy LP (finite-type model): expected counts y_{i,a} of authority-node types/modes are optimized subject to flow balance, a total risk-charge budget ∑ y_{i,a} r_{i,a} ≤ δ, and compute budget. Dual variables give decentralized shadow prices: λ (risk) and ν (compute), and imply threshold/base-stock rules for fanout decisions.
• Practical governor: immutably bind request + lineage; compute conditional certificate; atomically debit if allowance available; authorize exactly the specified action/scope; retries require new certificates.
• Synthetic numerical experiments illustrate the phase transition and option value of delayed vesting. Results are illustrative only; not claims about deployed-agent safety.

Data & Methods

• Formal model components:
  • Sandbox tree T_S: potentially countable adaptive generation of candidate branches with sandboxed (non-external) capabilities.
  • Activation sequence: chronological list of approved requests; for each activation i, history F_{i−1} (all pre-activation info) and event H_i (activation i causes catastrophe).
  • Certificate requirement: P(H_i | F_{i−1}) ≤ a_i; escrow enforces ∑ a_i ≤ δ.
• Theoretical results:
  • Theorem 1: uses conditional union bound (tower property + Boole) over adaptive sequence; monotone convergence handles countable sequences.
  • Proposition 1: compares reservation-accounting at spawn vs debit-at-activation; shows PRV weakly dominates by reclaiming allowance for unactivated spawns.
  • Theorem 2: stylized i.i.d. marked branching process. Authority-child-count generating function ψ(z)=φ(1−s+sz); fixed-point equation h=(1−p)ψ(h) gives no-harm probability. Taylor expansion of ψ around 1 yields asymptotic regimes; classical branching-process extinction arguments deliver the phase-transition behavior.
  • Occupancy LP (finite-type): linear program over expected mode occupancies y_{i,a} with flow-balance, risk budget, compute budget. Strong duality provides shadow prices (λ, ν) and continuation values V_i; decreasing marginal-value-per-risk assumption yields threshold fanout rules.
• Algorithm: A governor that enforces deterministic checks, computes selection-conditional certificate, performs atomic debit and authorization, and logs immutable binding between request and lineage.
• Empirics: two reproducible synthetic studies built from the models above to illustrate phase transition and option value. No field or deployment data; synthetic parameters chosen to demonstrate theoretical behavior.
• Key assumptions and required conditions:
  • Sandbox enforcement: sandboxed branches cannot cause defined catastrophe with their available capabilities.
  • Certificate validity conditional on the full pre-activation history (including any selection signals).
  • For branching asymptotics: i.i.d. per-node independence and finite second factorial moment where stated.
  • For LP results: finite types/modes, uniform subcriticality (spectral radius ≤ 1 − η), feasibility of LP.

Implications for AI Economics

• Internalizing risk with explicit prices: PRV turns a trajectory-level safety constraint into a scarce budget δ whose marginal value (shadow price λ) can be computed. This enables treating catastrophe-risk as an economic resource—priced and allocated—parallel to compute budgets (priced by ν). Firms can incorporate λ into product decisions, risk transfer, insurance pricing, and cost–benefit analysis.
• Value of exploration separated from exposure: PRV permits wide sandbox exploration at low cost while only charging realized authority-bearing operations. Economically, this increases optionality and reduces wasted reservation of scarce risk budget, improving expected productivity for a given safety constraint.
• Nonlinear systemic risk and regulation calibration: the authority-reproduction number R_A is a structural parameter determining whether small per-node improvements (lower p) meaningfully reduce system-level risk. If R_A > 1, reducing local failure probability can leave a residual floor of catastrophic risk—regulators and firms must cap R_A (limit fanout or s) as well as p. This argues for controls on delegation breadth in addition to per-node reliability standards.
• Design of incentives and contracting: shadow prices suggest how to compensate human approvers or third-party risk bearers, and how to structure internal quotas or markets for risk allowances (tradable δ allocations across teams or episodes). Threshold policies for fanout map to concrete operational rules (e.g., grant k children only if marginal value per unit risk ≥ λ).
• Governance & operational policy: PRV supports atomic, auditable authorization decisions and aligns incentives: branches must request authority and consume a measurable portion of the enterprise’s risk budget. This aids accountability, auditing, and fine-grained oversight costs modeling.
• R&D and strategic tradeoffs: firms choosing architecture (more recursive spawning vs stronger per-node reliability) face a tradeoff with nonlinearities. When near criticality, investing in slightly lower p can have outsized benefits (√p regime), but if supercritical, attention should focus on limiting R_A first. Budgeted risk pricing helps prioritize such investments.
• Limitations that affect economic modeling:
  • Guarantees rely on sandbox enforcement and conditional certificate validity; failures here create model mismatch and likely underpricing of systemic risk.
  • The stylized branching and independence assumptions simplify analysis; real systems may have dependencies that produce correlated failures and new floors.
  • Synthetic experiments illustrate mechanisms but do not estimate deployed-agent safety or real-world λ magnitudes.
• Practical policy takeaways for AI-economics stakeholders:
  • Implement explicit escrowed risk budgets (δ) as a governance primitive and measure their shadow price to guide allocation.
  • Allow broad sandboxed search (to preserve option value) but require selection-conditional certification and atomic debits before external effects.
  • Monitor and control authority reproduction (R_A) via fanout limits or per-activation acceptance probability s to avoid supercritical regimes that create irreducible systemic risk.
  • Use dual prices to design internal quotas, trading of risk allowances, and to prioritize investments in reliability vs. reduced delegation.

Summary recommendation (design rule): search broadly in the sandbox; vest authority sparingly and atomically, charging an explicit, selection-conditional risk allowance that is conserved across the episode.