A practical web-layer bargain for AI crawlers: terms.txt and a signed request/receipt exchange let origins enforce per-path, per-purpose access and price machine requests, and a dependency-free prototype shows enforcement and receipts are feasible with low measured latency in microbenchmarks.

Main Finding

The author documents that the web’s implicit crawler–publisher “bargain” has broken (automated clients now form a majority of requests and AI crawlers fetch far more pages per returned visitor than search), argues robots.txt is inadequate for modern machine access, and proposes terms.txt plus an origin-enforced HTTP exchange (built from Web Bot Auth, signed intent, delegation tokens, 402 negotiation, and signed receipts) to make per-path, per-purpose machine-access terms discoverable, enforceable at the origin, and auditable. A dependency-free prototype shows small per-request CPU costs (roughly 0.20–0.65 ms on a single vCPU in their tests), indicating practical deployability.

Key Points

• The bargain broke:
  • Automated clients were ≈51–57.5% of traffic (Imperva 2024–25, Cloudflare mid‑2026).
  • Crawl-to-refer ratios: conventional search ≈5:1; some AI platforms fetched orders of magnitude more pages per returned visitor (examples: Anthropic reported tens to hundreds of thousands per referral in some windows; mid‑2026 numbers fell but remained hundreds to thousands).
  • Training-related fetches rose substantially (Cloudflare-classified training requests 72% → 79% in 2024–25; broader 2026 data show training ~52% in classified requests; mixed‑use bots >36%).
• Robots.txt (RFC 9309) is insufficient:
  • Cannot express identity, purpose, price, delegation scope, or verifiable claims.
  • Not reliably honored (clients can spoof user-agent, rotate IPs).
  • Blunt policy causes crossfire (blocking training can block indexing/discovery).
• Existing components (standards + vendor features) cover parts of the problem:
  • Web Bot Auth + HTTP Message Signatures provide verifiable operator identity.
  • IETF AI Preferences (AIPREF) provides richer intent vocabulary but is by charter non‑enforcing.
  • Vendor solutions (Cloudflare Pay Per Crawl / Pay Per Use, Fairfetch) enforce pricing/authorization but are proxy/tunnel‑dependent and proprietary.
• terms.txt and the origin exchange (design highlights):
  • /.well-known/terms.txt is a robots.txt-style, per-path file declaring per-purpose rules (allow/charge/deny), use ceilings (e.g., reference vs full), payment endpoints, and receipt key locations.
  • Clients sign requests with Web Bot Auth (Signature-Agent, Signature-Input, Signature) and include Access-Intent, Access-Delegation, and Access-Payment headers/tokens.
  • The origin verifies identity, signature freshness/non-replay, delegation scope (pairwise pseudonymous subject bound to operator), and payment; then serves content and issues a signed Access-Receipt appended to an append-only, hash-chained receipt log.
  • Receipts prove what was delivered under which terms; they do not control what a model later does with bytes.
  • Delegation tokens and vouchers are operator-bound and pairwise pseudonymous to preserve user anonymity to the origin while conveying entitlement.
  • Cached responses are handled via private Vary on signature/intent/delegation so receipts cannot be reused across operators.
• Enforcement vs audit vs contract:
  • Enforced at request time: operator identity, signature freshness, signed intent, valid delegation, path-purpose terms, and payment presence where required.
  • Auditable after delivery: declared intent vs observed behavior (e.g., reproducing full content when intent was reference/search) via receipts, citations, and traffic.
  • Contractual/unenforceable by protocol: downstream use (e.g., whether a model actually trains on lawfully fetched content).
• Threat model and mitigations:
  • The protocol cannot stop false purpose claims—handled via audit and standing revocation.
  • Mitigations for credential theft and replay: short validity windows (minutes), atomic nonce reservation keyed to (operator URL, keyid, nonce), key rotation, rate limits, and pre-checks (expiry/nonce/key presence) before crypto.
  • Main systems risk: CPU cost for verification (mitigated by inexpensive checks first and refusal costing less than acceptance).
• Prototype and measured overhead:
  • Dependency-free Node.js (≈600 lines) prototype and test harness with end-to-end checks.
  • Loopback benchmarks on one 2.1 GHz Xeon vCPU (no TLS), headers pre-signed (client signing excluded):
  • Median Ed25519 verification ≈120 µs; delegation verify ≈122 µs; receipt signing ≈42 µs.
  • Measured added median latency over passthrough: unsigned allow path ≈+0.015 ms; identity-only requests ≈+0.201 ms; signed search (receipt + log) ≈+0.339 ms. Overall reported prototype overhead in the abstract: 0.20–0.65 ms per request on one vCPU.

Data & Methods

• Synthesizes public measurements from:
  • Cloudflare Radar (crawl-to-refer ratios, crawler categorization, automated-share metrics).
  • Imperva aggregated client-classification statistics.
  • Independent audits (e.g., Longpre et al. audit of /robots.txt across corpora like C4, RefinedWeb, Dolma).
  • Pew Research Center browsing/click experiments showing lower click-through when AI summaries present.
• Methods:
  • Comparative analysis of policy mechanisms (robots.txt, ai.txt/llms.txt, AIPREF, Web Bot Auth, vendor 402/payments).
  • Design and specification of terms.txt vocabulary and an HTTP-level exchange composed from IETF drafts and vendor conventions.
  • Prototype implementation in Node.js; loopback benchmarking with controlled concurrency, warm-up, and repeated runs; measurement of microsecond-scale crypto costs and millisecond end-to-end latencies.
• Reproducibility:
  • Code and raw results archived (DOI provided in preprint). The prototype tests numerous edge cases (delegation scope checks, replay/racing behavior, signature failures, receipt log integrity).

Implications for AI Economics

• Internalizing data-access externalities:
  • terms.txt + exchange makes access terms explicit, attributable, and measurable, enabling publishers to demand compensation (per-fetch or per-use) or deny purposes (training) without renting a vendor proxy.
  • Receipt logs provide auditable units of delivery; marketplaces/settlement services can build on those receipts to pay publishers or charge AI platforms.
• Shifts in pricing unit and incentives:
  • The market already moved from per-fetch to per-use (Cloudflare's Pay Per Use experiments). This protocol decouples unit choice from enforcement, allowing the market (publishers, platforms, intermediaries) to settle the unit of account.
  • If publishers charge for training but allow search/agent reference, AI providers face increased marginal costs to assemble training corpora, potentially raising barriers to entry for smaller models or shifting business models toward licensed datasets.
• Discovery vs training trade-off:
  • Mixed-use crawlers (search + training) create an inherent tension; terms.txt lets publishers express differentiated policies per purpose, but platforms with dominant search reach (e.g., Google) may have leverage—raising distributional and competition concerns.
• Operational feasibility and deployment path:
  • Low measured per-request CPU overhead suggests origins can adopt the exchange without major performance penalties, making origin-side enforcement realistic.
  • However, broad economic impact requires wide adoption across browsers, agent operators, identity and settlement providers, and CDNs; vendor defaults (e.g., Cloudflare) will shape the initial landscape and could concentrate power.
• Residual risks and limits:
  • Protocol enables monetization and auditability but cannot by itself prevent non-compliant scraping or post-delivery uses (training); enforcement is social/legal/contractual after audit.
  • Evasion (unsigned scraping, forged headers, proxyed fetches), platform noncompliance, and concentration of intermediaries remain policy and market risks.
• Policy and market recommendations implied by the work:
  • Standardize and interoperate on identity (Web Bot Auth), intent vocabularies (AIPREF), and receipt/settlement primitives to enable multivendor markets for data access.
  • Build settlement infrastructure that consumes receipts and supports per-use or per-answer remuneration models.
  • Monitor and regulate vendor default settings to avoid unilateral lock-in where one CDN sets de facto global terms.

Summary takeaway: terms.txt plus an origin-enforced, signed-request exchange is a technically lightweight, auditable way to make machine-access terms and payments discoverable and enforceable at the origin. It does not eliminate misuse or substitute for contracts and law, but it creates the measurement and attribution primitives needed to internalize web access externalities and to enable economic arrangements between publishers and AI platforms.