LLM agent memories can invent authority and induce unauthorized actions: a benchmark shows typed incremental memories create false permissions in up to ~50% of cases and executors act on them in ~99% of trials, while oracle memory repair prevents all such violations.

Main Finding

Endogenous authorization laundering: an LLM agent’s persistent memory can itself create spurious permissions that the true history never granted, and those false permissions almost always lead downstream executors to perform unauthorized actions. The authors introduce EAL-BENCH to measure this effect, show it arises across domains and model families (without any attacker), and demonstrate that memory-level interventions (e.g., exact-state repair, provenance gating, or event-sourced maintenance) are required to stop it — but these defenses trade safety for lost legitimate actions.

Key Points

• Definition and decomposition
  • Formation (F): memory encodes authority that the canonical ledger denies.
  • Propagation (G): the executor acts on that memory-encoded authority.
  • End-to-end laundering = F ∧ G.
• Measured scale
  • Typed incremental memory produced false authority in up to 50.2% of unauthorized requests (finance).
  • Once false authority was present, executors acted on it in 98.6% of matched trials (i.e., nearly all formed false permissions propagated to unauthorized action).
  • Exact-state repair (replacing erroneous memory with oracle-correct state) eliminated unauthorized actions in all tested cases (0% after repair), isolating the failure to memory formation.
• Memory design matters
  • Incremental updating (Mt = W(Mt−1, Bt)) is much more failure-prone than one-shot reconstruction from full history.
  • Typed memories enable deterministic detection of formation (auditable before execution) but are not by themselves sufficient to prevent formation.
  • Free-text memory cannot be deterministically evaluated for formation, so causality is established via memory-replacement experiments.
• Mitigations and tradeoffs
  • Two defenses evaluated: (1) require stored permissions to cite valid authoritative sources (gold cited-source gating), and (2) use bounded event sourcing (extract changes per block into an immutable log and deterministically reduce).
  • Both substantially reduce laundering but increase undergranting (legitimate actions being rejected), exposing a safety–utility tradeoff.
• Generality
  • Effect observed across three domains (procurement, cybersecurity, finance), five writer models, and two executor models — the failure is writer-general and the erroneous memory artifact carries the problem across executors.

Data & Methods

• Benchmark: EAL-BENCH
  • Each case: multi-session organizational history H (sequence of blocks with grants, amendments, revocations, non-authoritative records) + a hidden deterministic canonical ledger R that yields the true authorization state St.
  • For each case, a matched pair of requests (one authorized, one unauthorized) differing only in the authorization-relevant field.
• Memory treatments
  • Representations: free-text (single Markdown-like string) vs typed (schema-validated JSON authorization records with source identifiers).
  • Update strategies: one-shot (writer sees full history and creates MT) vs incremental (writer updates MT from previous memory plus new block, never re-reading raw past blocks).
  • Memory capacity bounded (calibrated) and writers allowed limited validation/repair attempts.
• Models evaluated
  • Writers (memory creators): Nemotron 3 Ultra, Kimi K2.6, GLM 5.2, Grok 4.3, Qwen-Plus.
  • Executors (actuators): GPT-OSS-120B and DeepSeek V4 Pro.
  • All models run with fixed parameters; three random seeds used for replication.
• Metrics & interventions
  • Authorized use: fraction of authorized requests where executor performs the action.
  • Unauthorized submission: fraction of unauthorized requests where executor performs the action.
  • Formation P(F) computed deterministically for typed memories; free-text assessed by replay interventions.
  • Interventions: exact-state repair (swap in oracle memory), pressure messaging (adds urgency without new authorization), mitigation gates (cited-source validation), bounded event sourcing.
• Key quantitative results (selected)
  • Typed incremental memory: P(F) = 28.3% (procurement), 10.4% (cybersecurity), 50.2% (finance); corresponding unauthorized submissions roughly match these rates.
  • Exact-state repair: unauthorized action rates dropped from ~98% under erroneous memories to 0% under oracle memory (per-domain counts in paper).
• Reproducibility
  • Open-source benchmark, run manifests, and retained artifacts; experiments pooled across seeds for descriptive replication (not precise inference).

Implications for AI Economics

• Memory is part of an agent’s effective authorization policy
  • Persistent memory design and maintenance directly determine whether an agent can legally/contractually act; this converts an engineering/performance decision into an economic and governance one.
• Endogenous risk raises expected loss and deployment cost
  • Unauthorized purchases, trades, or system changes caused by memory errors produce direct financial losses, regulatory risks, and operational disruptions even without external attack. Expected liability increases for firms deploying long-running agents.
• Auditing and insurability
  • Typed memories that allow deterministic formation checks enable pre-execution audits (estimate P(F) before granting tool access) — improving insurability and risk management. Free-text memories are harder to audit.
  • Insurers and underwriters should consider memory-design and update strategy (incremental vs one-shot vs event-sourced) when pricing coverage.
• Safety–utility tradeoffs affect adoption and productivity
  • Provenance gating / event-sourced designs reduce false authorizations but raise false negatives (undergranting), imposing operational friction and labor costs (more human approvals or delays).
  • Firms must optimize between risk-reduction (lower unauthorized action rate) and throughput/productivity (higher authorized use).
• Organizational and market design responses
  • Protocols, standards, and certification for agent memory management (e.g., required provenance metadata, event-sourcing standards, bounded replay intervals) could reduce systemic risk and lower compliance costs.
  • Contracts and liability allocation (vendors vs deployers) need to account for endogenous laundering risk: vendors may be liable if default memory-management practices are unsafe; buyers may require particular memory architectures as procurement conditions.
• Policy and regulatory considerations
  • Regulators may require auditable provenance, deterministic state reconstruction, or minimum guarantees on false-authorization rates for automated agents acting in regulated domains (finance, procurement, critical infra).
  • Disclosure requirements about memory design and audit logs could be mandated for high-stakes deployments.
• Recommended near-term practices (economic/practical)
  • Prefer auditable typed representations for authorization-sensitive agents.
  • Avoid purely incremental-only memory updates for long-running authorization state, or schedule periodic full-history reconciliation.
  • Invest in deterministic event-sourced memory maintenance or external provenance verification to shift trust from LLM heuristics to verifiable processes.
  • Measure P(F) as a risk metric and include it in cost–benefit and compliance analyses when deciding to automate tasks that affect monetary flows or permissions.

Summary: The paper demonstrates a sizable, systemic, non-adversarial source of economic risk from how LLM agents store and update persistent memory. Memory management choices materially affect expected unauthorized actions and thus should be treated as first-order economic, legal, and governance variables when deploying LLM agents in finance, procurement, cybersecurity, and other high-stakes domains.