Compile meaning once, read cheaply: ingest-time semantic compilation (validated, claim-level payloads plus maintained embeddings) answers dialogue questions more accurately while using far fewer reader tokens, and incremental upkeep is tens of times cheaper than full rebuilds; only a heavy contextualization+rerank stack matches accuracy at roughly 21× the per-query token cost.

Main Finding

Ingest-time semantic compilation (ISC) — compiling a corpus’s meaning into a maintained, queryable semantic substrate at write time — outperforms the dominant pattern of re-interpreting raw text on every query (query-time semantic reconstruction, QSR). ISC (1) makes reads much cheaper per query by paying once at ingest and maintaining the result incrementally, (2) yields higher reading accuracy per token (or the same accuracy for far fewer tokens), and (3) can be cost-amortized for realistically queried corpora. The paper demonstrates both a geometric existence proof (cheap, precise incremental maintenance of embeddings) and an applied retrieval-QA proof (compiled, provenance-validated claims as retrieval payloads dominate chunk-based baselines).

Key Points

• Paradigm: ISC stores two coupled layers as a first-class database object:
  • Geometric layer: incrementally maintained embeddings and their index.
  • Symbolic layer: atomic, provenance-validated claims (claim text + exact source span, speaker, offsets).
• Validation gate: candidate claims must carry a verbatim source span located in the canonical transcript; unsupported (hallucinated) claims are rejected at write time.
• Maintenance scales with change, not corpus size: incremental low-rank updates are much cheaper than full re-decomposition and can absorb embedding-model upgrades via alignment (orthogonal Procrustes).
• Read-time performance:
  • Compiled claims achieved 85.2% accuracy at a ~2,048-token reader budget with roughly 2.2k reader tokens.
  • Best chunk-based configuration reached 72.5% at ~16.3k reader tokens.
  • A sophisticated contextualized-chunk stack (hybrid retrieval + reranker + preambles) matched facts on accuracy (~88% vs 85.2% in some cells) but required ~21× the query-path tokens (≈47.7k vs ≈2.2k).
  • Compiled claims carried gold evidence in 98–99% of cases; chunk reading accuracy degraded as more raw context was supplied.
• Cost amortization: compiling the studied corpus (~500 transcripts) was estimated at ≈26.3M ingest tokens (~$32 at hosted pricing in the replay) and amortizes within the first few hundred–thousand reads for that workload. The paper frames a break-even read count R* as a planner statistic deciding whether to compile.
• Research agenda: compilation planners (per-document R*), provenance-as-integrity-constraint design, production-scale maintenance & migration, multi-tenant substrate economics, read planning (how much compiled context to read), evaluation metrics for compilers, and interactions with agentic systems.

Data & Methods

Two complementary existence proofs:

1) Synthetic geometric pilot (embedding maintenance) - Corpus grown from 3,000 → 9,000 synthetic documents over 50 update events, constructed to isolate maintenance cost. - Incremental low-rank updates: 8.4 ms per update vs 283 ms for full re-decomposition (≈33.7× cheaper per update; ≈23.8× cheaper cumulatively). - Numerical fidelity: principal-angle drift < 1e-11 degrees; recall@10 = 1.0. - Embedding-model migration handled with orthogonal Procrustes: recovered 0.95 mean cosine similarity while re-embedding only ≈10% of corpus.

2) Retrieval-QA on real dialogue transcripts (symbolic payloads) - Corpus: broadcast-interview transcripts. Held-out sample: 500 transcripts and 499 questions (separate development set used to build extractors). - Compiled-substrate implementation: PostgreSQL tables for facts and fact_evidence; strict UNIQUE(document_id, content_hash) for idempotence; exact-quote string-match validation gate. - Validation outcomes: the gate admitted 69,746 claims in the corpus; in a 20-document replay it rejected 1.1% (29/2,724) of candidate claims, mostly due to spans not located verbatim in the source. - Baselines: fixed-width chunking, turn-aware chunks, semantic chunks; a contextualized-chunk stack with hybrid dense+full-text retrieval (reciprocal rank fusion) plus cross-encoder reranker was a pre-registered, strong baseline. - Retrieval/answering experiment: identical embedding, retrieval, and reader models across conditions; comparisons run across 32 budget-by-model cells; paired per-question outcomes tested with McNemar’s test and Holm correction for multiple comparisons. - Key performance numbers: - Compiled claims: 85.2% correct at 2,048 token budget (≈2.2k reader tokens). - Best chunk config: 72.5% correct at ≈16.3k reader tokens. - Contextualized-chunk stack: ≈88.0% correct at ≈47.7k reader tokens; statistically indistinguishable from compiled claims in the strongest cell (McNemar p = 0.202), but consumed ~21× tokens. - Cost estimate for compilation (replay): extraction averaged 48.2k prompt + 4.5k completion tokens per document ≈ $0.064/doc; compiling 500 docs ≈ 26.3M tokens and ≈$32 under the replay pricing assumptions. Given the contextualized stack’s extra ~45.5k tokens per query, compilation cost ≈ the token cost of ~580 queries, so amortization occurs within the first ~1k reads for this corpus. - Limitations noted: synthetic pilot idealized; grading used the same model family as extraction (automated judgment); human calibration and cross-model rejudgment are in progress; domain limited to broadcast transcripts.

Statistical evidence summarized: - Facts beat chunk baselines in all 32 budget-by-model cells; all 24 facts-versus-chunk comparisons survived Holm correction (p ≤ 4.9×10^-6; strongest 1.4×10^-42). - Only the assembled contextualized stack could match facts on accuracy, but at much higher token cost.

Implications for AI Economics

• Shift in compute expenditure: ISC trades repeated per-query inference cost (expensive, especially as context size grows) for one-time or incremental ingest-time compute. For frequently-read corpora, this lowers marginal cost per query substantially and can change the cost structure of deployed RAG systems.
• Break-even planning (R*): whether to invest in compilation becomes an explicit economic decision. Systems should model expected read volume, document volatility, and compilation cost to decide per-document what to compile and to what depth (embeddings only, claims, richer relations).
• Billing and pricing models:
  • Current token-based billing (per-model prompt/completion) may underprice or misalign incentives: providers get paid more for repeated costly reads; ISC reduces per-query tokens and thus potentially reduces variable revenue. This could motivate new pricing (e.g., subscription/flat fees for compiled substrate maintenance, per-ingest billing, or charges for maintenance and migration operations).
  • Vendors might offer “compiled-substrate as a product” (managed maintenance, migration, access controls) with distinct billing and SLA tiers.
• Vendor and market effects:
  • Demand for compiler/maintenance tooling, substrate index services, and migration/alignment tooling will grow, creating new product categories.
  • Multi-tenant compiled substrates create positive externalities: many agents can reuse the same compiled semantics, improving economic efficiency for organizations but concentrating value in substrate providers—raising competition and access concerns.
• Model-product design choices:
  • With ISCs, teams can rely on cheaper reader models and smaller-context readers because the payload is already distilled and validated, reducing the need to pay for the highest-context frontier LLMs on every read. This can materially reduce operating costs for high-throughput systems.
  • However, embedding-model churn and migration impose maintenance and alignment costs that must be priced into long-term budgets.
• Incentives and welfare:
  • ISC increases returns to investing in quality ingest-time extraction (better validators, stricter provenance rules) because errors are persistent; this raises the cost of mistakes but aligns incentives toward robust validation.
  • Systems that continue to QSR externalize repeated inference costs onto users/tenants and may be economically inefficient for hot data; suppliers of ISC tooling could capture much of the surplus.
• Risks and externalities:
  • Upfront capital cost: small or volatile corpora might never recoup compilation costs; planners must prevent over-investment.
  • Concentration of knowledge assets: compiled, validated substrates could become proprietary assets with lock-in effects.
  • Privacy/access control and multi-tenant economics require new mechanisms (per-claim access control, versioning, isolation), which have economic implications for pricing and regulatory compliance.
• Research-to-product pathway: empirical measures (per-document read curves, maintenance cost per change, validation-rejection rates) must become first-class inputs to cost models and optimizers; marketplace designs and billing primitives should evolve to reflect one-time + maintenance + per-read price decomposition.

Short takeaway for practitioners and economists: ISC reframes the dominant cost tradeoff in retrieval-augmented systems from “pay many times for expensive semantic interpretation” to “pay once (or incrementally) to compile validated semantics, then read cheaply.” For many workloads with repeated reads on relatively stable documents, that is a strictly better economic model — but it requires new tooling, billing primitives, and planner logic to decide what to compile and when.