A formal impossibility: when combined human-AI autonomy exceeds a calculable threshold, responsibility cannot be fully and legitimately assigned to people without reducing system autonomy; simulations on 3,000 synthetic collectives confirm a sharp phase transition.

Main Finding

The paper proves a formal impossibility: in a mathematically specified class of Human-Agent Collectives (HACs), no single-locus accountability framework can simultaneously satisfy four minimal legal/philosophical axioms (Attributability, Foreseeability Bound, Non‑Vacuity, Completeness) whenever the collective (i) contains a mixed feedback cycle involving both humans and artificial agents and (ii) its minimum compound autonomy exceeds a computable threshold (the "Accountability Horizon") ˆΛ* = 1 − 1/|C_min|. Below that threshold, legitimate single-locus accountability frameworks exist; above it, the impossibility is structural — transparency, audits, or oversight cannot restore the four axioms without reducing agent autonomy.

Key Points

• Formalisation: The paper introduces Human-Agent Collectives (HACs) as an SCM (structural causal model) of joint human-AI systems. Agents (human or artificial) are state-policy tuples with explicit observation spaces; interactions are described by an interaction graph and joint action/outcome mapping.
• Four-dimensional autonomy profile: Each artificial agent has a computable autonomy vector α = (αE, αX, αD, αS):
  • Epistemic (αE): informational independence of agent beliefs from supervisors;
  • Executive (αX): propensity to act outside human approval;
  • Evaluative (αD): divergence of agent utility from human utility;
  • Social (αS): degree of unsupervised inter-agent initiation. Aggregate autonomy A(a) = w⊤α(a) with positive weights w.
• Mixture model assumption: Artificial agent policy is a convex mixture of a human-aligned component and an autonomous component; executive autonomy αX is the mixture weight.
• Accountability axioms (single-locus accountability):
• Attributability — responsibility must be causally grounded (NESS-style).
• Foreseeability Bound — an agent’s responsibility cannot exceed its predictive/epistemic capacity.
• Non‑Vacuity — at least one agent must bear non-trivial responsibility.
• Completeness — total responsibility for an outcome must be exhaustively allocated across agents.
• Accountability Incompleteness Theorem: If the minimum compound autonomy (product of executive and epistemic autonomy for agents in a minimal mixed feedback cycle) exceeds ˆΛ*, then no allocation mechanism can satisfy all four axioms simultaneously. The structural source is distributed, feedback-rich autonomy; shifting information or transparency does not eliminate the gap unless autonomy is reduced.
• Phase transition: The theorem establishes a sharp boundary — below ˆΛ* feasible single-locus allocations exist; above it they do not.
• Empirical validation: Computational experiments on 3,000 synthetic HACs matched theoretical predictions with zero violations under the model.
• Robustness: Result tolerates approximate (TV-bounded) deviations from exact mixture structure; choices of positive weight vectors w do not affect the theorem; human agents are modelled in maximally favourable way (weakening human model only strengthens the result).

Data & Methods

• Formal model:
  • Environment E = (Ω, O, T) with shared and exogenous observation components.
  • Agents a = (S, A, Ω_i, δ, ω) where S = internal states, A = actions, Ω_i = observations, δ = state transitions, ω = policy.
  • Collective SCM M = (U, V, F, P(U)) following Pearl: endogenous variables include agent states/actions and outcome O; exogenous include environment state and noise.
• Autonomy metrics: Information‑theoretic definitions (mutual information and entropy) for epistemic autonomy; probability-based definition for executive autonomy; normalized utility distance for evaluative autonomy; communication counts for social autonomy. Compound autonomy used multiplicatively for executive × epistemic (αX·αE).
• Structural assumption: Agent policies adhere (exactly or approximately) to a convex mixture of human-aligned policy π_h and autonomous policy π_g that uses private information.
• Accountability axiomatization: Four axioms motivated by legal/philosophical literature (NESS causal tests, foreseeability in tort/moral responsibility, non-triviality, and exhaustiveness).
• Mathematical results: Lemmas and the main Accountability Incompleteness Theorem proved in the SCM framework; corollaries characterize accountability residuals and existence of allocations below horizon.
• Computational validation: 3,000 synthetic HACs sampled under the model; tests confirmed theoretical boundary and phase transition; reported zero empirical violations.
• Supplementary results: Propositions on δ-robustness (total variation perturbations) and sensitivity of the Accountability Horizon to modelling noise.

Implications for AI Economics

• Regulatory boundary and calibration:
  • The paper provides a computable criterion (Accountability Horizon ˆΛ*) that regulators can use to distinguish systems where traditional single-locus liability is structurally feasible from systems where it is impossible. This enables risk-tiering and more precisely targeted obligations (e.g., require reduced autonomy or distributed governance above the horizon).
• Firm incentives and product design:
  • Firms choosing to deploy agentic, feedback-rich multi-agent systems face an autonomy–accountability trade-off with economic consequences: maintaining high autonomy may remove the possibility of assigning liability to a single actor, increasing legal uncertainty, reputational risk, and transaction costs (insurance, contract complexity). Firms will need to decide whether to reduce autonomy (incurring performance costs) or adopt new governance/contractual structures (coalitions, joint‑and‑several liability, pooled insurance).
• Liability, insurance, and financial markets:
  • Traditional liability markets and insurers price risk assuming accountable counterparties. Above the Accountability Horizon, residual unallocatable responsibility implies either (i) higher premiums, (ii) new insurance products for collective/coalition liability, or (iii) market failure for some applications. Insurers, investors, and corporate risk managers must factor in whether deployed systems cross the horizon.
• Market structure and competition:
  • Smaller suppliers or firms that deliberately constrain autonomy to remain below ˆΛ can credibly offer accountable products at potentially lower legal/regulatory cost. Conversely, firms that push autonomy above ˆΛ may capture performance gains but face higher governance costs and regulatory scrutiny, which can lead to concentration if incumbents better absorb such costs or create proprietary distributed-governance institutions.
• Social cost of autonomy vs. innovation:
  • The theorem quantifies a structural externality: high-autonomy networks can create accountability voids that impose social costs (uncompensated harms, uncertainty). Policymakers can use this to weigh innovation benefits against governance externalities and design incentives (subsidies for accountable design, taxes on opaque autonomy).
• Policy design recommendations (economic levers):
  • Mandate design limits or verifiable contracts that keep effective αX·αE below threshold for high‑risk applications.
  • Encourage polycentric/distributed accountability (coalitions, joint liability, statutory collective responsibility regimes) and standardize mechanisms for collective allocation of responsibility.
  • Develop new insurance primitives and disclosure requirements for collective risk (e.g., mandatory registries of interaction graphs, minimal observability standards).
  • Use the Accountability Horizon as a quantitative metric for regulatory classification (risk tiers) and as a compliance target businesses can audit.
• Research & market implications:
  • Need for economic models of contracting in the presence of non‑allocable responsibility (how do firms contract, share revenue and liability when some harms cannot be singularly attributed?)
  • Potential growth of third-party governance markets: firms offering governance-as-a-service (distributed accountability solutions, coalition formation, monitoring contracts) to manage exposure above the horizon.
  • Implications for valuation and M&A: acquiring a firm whose systems sit above ˆΛ* carries additional governance liabilities; diligence should include autonomy-profile audits.

Limitations to bear in mind - The impossibility is proved inside a specific formal model and depends on modelling choices (SCM, autonomy metrics, mixture structure, completeness axiom). Stakeholders may dispute normative axioms (e.g., rejecting Completeness). - Threshold ˆΛ* depends on |C_min|, the minimal size of a mixed human–AI feedback cycle; practical application requires mapping real-system topology into the model and estimating autonomy metrics. - The human agents in the analysis are modelled favourably; relaxing that assumption generally strengthens the impossibility (i.e., makes governance harder), but real-world complexities (continuous time, learning dynamics, legal institutions) require applied calibration.

Bottom line The paper gives AI economics a formal, computable boundary — the Accountability Horizon — that separates regimes where traditional single‑actor accountability is feasible from regimes where it is provably impossible. This has direct consequences for regulation design, firm strategy, insurance markets, and the economics of product design and governance infrastructure.