0 cumulative citations
View corpus contextA cache-aware compression policy halves LLM API bills: by compressing prefixes in a cache-preserving way, CAPC preserves hot-tier cache hits and cuts costs substantially across benchmarks and three production workloads, delivering large savings with negligible quality loss.
Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
Cache-Aware Prompt Compression (CAPC) combines query-agnostic compression with cache-control and a tier-preserving ratio bound to preserve cache hits, yielding large API cost reductions (median savings ~49% vs cache-only, 64% vs query-aware compression, up to 90% vs vanilla) while keeping task quality nearly unchanged.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios (r>=6). We propose Cache-Aware Prompt Compression (CAPC), pairing query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix (51.7% cost reduction at r=3); a graphify knowledge-graph RAG pipeline across two codebases (9.3x vs cache-all on FastAPI, 2.4x on httpx); and the public tau-bench retail benchmark (50 tasks), where CAPC is the cheapest of four strategies with reward exactly equal to vanilla (both 36/50, p=1.00) while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.
Summary
Main Finding
Prompt-compression and prompt-caching interact nontrivially under real-world API caching. Anthropic Sonnet 4.6 exposes a two‑tier, size-dependent cache: prefixes below ≈3.5k tokens live in a “hot” tier with hit rate plateau ρ ≈ 0.83, while larger prefixes hit persistently at ρ ≈ 1.0. Under realistic ρ(N, |P|) the dominant literature assumption ρ = 1.0 is misleading: (a) query-aware compression (which produces a different compressed prefix per query and thus breaks prefix caching) can outperform naïve cache-only at high compression ratios (r ≳ 6); (b) a simple Cache-Aware Prompt Compression (CAPC) strategy — query-agnostic compression + cache_control with a tier-preserving ratio bound — strictly dominates baselines across workloads tested, delivering large cost savings at near-baseline quality.
Key Points
- Two-tier cache empirics (Sonnet 4.6, May 2026):
- Threshold T ≈ 3,500 tokens separating hot (smaller) and persistent (larger) tiers.
- Hot-tier plateau: ρ ≈ 0.83 (stable across trials); persistent tier: ρ ≈ 1.0 from the first subsequent call.
- Token-strict prefix invalidation (with a tokenizer caveat: leading/trailing whitespace normalized).
- Provider pricing on Sonnet 4.6 used in paper:
- pin (uncached input) = $3.00 / Mtoken
- cw (cache write) = $3.75 / Mtoken (5-min TTL example)
- cr (cache read) = $0.30 / Mtoken
- Dimensionless: α = cw/pin = 1.25, β = cr/pin = 0.10.
- Cost model (per-call, dropping common query/output terms):
- For cache-only vs compressed cached prefix, the crossover hit-rate threshold is ρ_cross(r) = (cw − pin / r) / (cw − cr).
- With Sonnet prices, ρ_cross grows with r; at r ≥ 6 the required ρ_cross > empirical hot‑tier plateau, so query-aware compression (which destroys cache) becomes cheaper than cache-only in practice for high r.
- CAPC (Cache-Aware Prompt Compression):
- Compress document once with a query-agnostic compressor, store the compressed block under cache_control, send dynamic query blocks per call.
- Add a tier-preserving ratio bound to avoid pushing the cached prefix into the hot tier: r_max_safe(D) = floor(|D| / T) (T ≈ 3,500 tokens for Sonnet 4.6).
- AdaptiveCacheBoundary extension: classify sentence positions as STATIC / QUASI / DYNAMIC by fingerprinting K versions; construct maximal STATIC-or-QUASI prefix for caching.
- Empirical dominance and savings:
- LongBench-v2 (16 doc-size × ratio configurations): CAPC cheapest in all 16 cells.
- Mean savings vs cache-only: 49% (range 24–67%)
- Mean savings vs query-aware compression: 64% (range 42–76%)
- Mean savings vs vanilla: 90% (range 84–94%)
- Quality within 0.05 of uncompressed baseline at tier-preserving ratios.
- Production validations:
- Enterprise tool-using assistant (94k-token tools= prefix): 51.7% cost reduction at r = 3.
- Graphify knowledge-graph RAG replicated on two codebases:
- FastAPI: 9.3× cost improvement vs cache-all
- httpx: 2.4× improvement
- Cache hit rate stable ≈85% across 5k–260k prefix sizes in those runs.
- τ-bench retail (50 deterministic tasks): CAPC cheapest and task-completion reward equal to vanilla (36/50); query-aware compression was +40.1% more expensive than vanilla.
- LongBench-v2 (16 doc-size × ratio configurations): CAPC cheapest in all 16 cells.
- Spend and reproducibility details:
- All Anthropic API experiments cost $98.96 total.
- Measurements used RUN_NONCE to avoid cache pollution; billing reconciled to provider rates within <1%.
Data & Methods
- Platform: Anthropic claude-sonnet-4-6, ephemeral 5-minute TTL caching.
- Empirical characterization:
- Prefix size sweep: {2k, 4k, 6k, 8k} token prefixes, n = 3 independent trials per size, N up to 30 (and separate runs up to N = 50 at |P| ≈ 2.4k).
- Measured cumulative cache hit rate ρ(N, |P|) and observed sharp step between 2k and 4k prefixes.
- Tokenization behavior tested with controlled mutations (start/mid/end edits) and found token-strict invalidation except for tokenizer normalization of leading/trailing whitespace.
- Experiments:
- LongBench-v2: documents 12k–25k tokens, compression ratios r ∈ {2,…,6}, N = 10 queries per cell, 16 total configurations.
- Enterprise assistant: 94k-token prefix experiment at production scale.
- Graphify RAG integration: two replicated codebases with different model priors to assess robustness.
- τ-bench retail: 50 deterministic database-state reward tasks for judge-free ground-truth.
- Implementation details:
- CAPC compresses once off the request path and reuses compressed prefix for all queries.
- AdaptiveCacheBoundary computes normalized fingerprints per sentence position across K versions, uses thresholds ε_static = 0.05, ε_quasi = 0.30 to pick cached prefix.
- Tier-preserving bound prevents over-compression that drops compressed size below T.
Implications for AI Economics
- Measurement-first cost modeling matters: treating cache hit rate as ρ = 1.0 can drive wrong design choices. Providers' caching architecture (two-tier vs uniform) materially alters the ROI of compression strategies.
- Design rule for practitioners: measure ρ(N, |P|) for your provider and compute ρ_cross(r) = (cw − pin / r) / (cw − cr) using your prices (or equivalently α, β). If your measured ρ for the compressed prefix falls below ρ_cross(r), query-aware compression (despite breaking cache) may beat cache-only; otherwise prefer cache-preserving compression.
- Value of precomputation: Query-agnostic compression makes compression precomputable and cache-preserving, unlocking large recurring savings when prefixes are reused. CAPC shows large practical savings with negligible quality loss when applied with the tier-preserving bound.
- Pricing and provider design incentives:
- Provider-side parameters (write premium α, read discount β, cache architecture) strongly influence optimal client-side tactics. Providers might respond by changing caching policies (e.g., semantic matching) or pricing, which would shift the crossover boundaries.
- If providers adopt semantic/robust prefix matching, query-aware compression could recover cache hits and narrow CAPC’s lead — but CAPC still benefits from token reduction.
- Operational recommendations:
- For deployments with large static prefixes and repeat queries, use query-agnostic compression + cache_control and enforce a tier-safe compression ratio to avoid reintroducing write-heavy hot-tier costs.
- Use an AdaptiveCacheBoundary for evolving documents to maximize cached static content without caching highly dynamic fragments.
- Re-run the small empirical characterization (RUN_NONCE, measure ρ(N, |P|)) after provider changes or model upgrades — the framework's formulas let teams recompute safe ratios and crossover points quickly.
- Broader economic effects:
- CAPC-like practices can materially reduce provider revenue per active user for workloads with large reusable prefixes; providers may respond via pricing changes, TTL adjustments, or smarter caching mechanisms (which in turn will change client-side optimal strategies).
- The paper provides a portable framework (cost model + empirical characterization method) so organizations can translate provider-specific prices and caching behavior into concrete operational decisions and expected savings.
If you want, I can: - compute ρ_cross(r) for your provider given your pin, cw, cr; or - produce a short implementation checklist and minimal code snippet to integrate CAPC + AdaptiveCacheBoundary into an existing request pipeline.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper uses direct measurements on a production LLM API, a simple cost model whose predictions are validated across benchmarks and multiple production workloads, and reports clear, large effect sizes; however evidence is limited to a single provider/model (Sonnet 4.6) and depends on provider-specific cache architecture and threshold behavior, so external validity to other models/providers is untested.
Methods Rigorhigh — Authors combine measured system telemetry (cache hit rates across sessions), an explicit analytic cost model, ablation-style comparisons of competing strategies (cache-only, query-aware compression, CAPC, vanilla) across a standard benchmark (LongBench-v2) and three production workloads, and report quantitative savings and fidelity metrics; limitations include potential absence of randomized assignment, limited provider diversity, and possible sensitivity to workload selection.
SampleEmpirical measurements on Anthropic Sonnet 4.6 API (30-call sessions, measured cache hit rates and a observed two-tier cache with a ~3,500-token threshold); experiments on LongBench-v2 across 16 configurations; three production workloads: an enterprise assistant with a 94k-token schema prefix, a graphify RAG pipeline across two codebases (FastAPI and httpx), and the public tau-bench retail benchmark (50 tasks). Compression ratios and quality (reward/fidelity) metrics are reported for each configuration.
Themesproductivity adoption org_design
IdentificationControlled empirical benchmarking combined with a cost model: measure real-world cache hit rates on Anthropic Sonnet 4.6, derive a crossover cost model that predicts when query-aware compression loses to caching, and validate predictions via controlled experiments on LongBench-v2 and three production workloads; no formal causal identification framework (e.g., randomization) beyond comparative experiments.
GeneralizabilityResults are based primarily on Anthropic Sonnet 4.6 and its cache architecture; other providers/models may have different caching policies and thresholds., Observed ~3,500-token threshold and rho≈0.83 plateau may not hold for other APIs, model sizes, or future provider implementations., Workloads include very large static prefixes (e.g., 94k-token schema); smaller or different real-world prefixes could change the relative benefits., Quality-preservation claims rely on the chosen quality metric (reward within 0.05); other tasks or stricter metrics may show different trade-offs., Operational constraints (latency, privacy, ability to set cache_control) and pricing models (per-token vs tiered billing) vary across deployments and could affect savings.
Claims (13)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Organizational Efficiency | null_result | cache hit rate (rho) |
Reading fidelity
high
Study strength
medium
|
n=30
rho~0.83
|
| Caching is far from the rho=1.0 ideal the literature assumes (observed rho ~0.83). Organizational Efficiency | negative | cache hit rate (rho) compared to ideal |
Reading fidelity
high
Study strength
medium
|
n=30
rho~0.83 vs ideal rho=1.0
|
| Under realistic rho, query-aware compression beats naive caching at high compression ratios (r >= 6). Organizational Efficiency | positive | cost (price per query / cost reduction) comparing query-aware compression and naive caching |
Reading fidelity
high
Study strength
medium
|
r >= 6 condition
|
| Cache-Aware Prompt Compression (CAPC) pairs query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. Organizational Efficiency | positive | prevention of cache promotion into hot tier (operationally: maintaining cache tier and cost savings) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2. Organizational Efficiency | positive | cost (cheapest strategy count across configurations) |
Reading fidelity
high
Study strength
medium
|
n=16
cheapest in 16/16 configurations
|
| On LongBench-v2, CAPC achieves mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla. Organizational Efficiency | positive | mean cost savings relative to alternative strategies |
Reading fidelity
high
Study strength
medium
|
n=16
49% over cache-only; 64% over query-aware; 90% over vanilla
|
| CAPC achieves output quality within 0.05 of the uncompressed baseline on LongBench-v2. Output Quality | null_result | output quality (difference from uncompressed baseline) |
Reading fidelity
high
Study strength
medium
|
n=16
within 0.05 of uncompressed baseline
|
| On an enterprise tool-using assistant with a 94k-token schema prefix, CAPC yields a 51.7% cost reduction at r=3. Organizational Efficiency | positive | cost reduction |
Reading fidelity
high
Study strength
medium
|
n=1
51.7% cost reduction at r=3
|
| In a graphify knowledge-graph RAG pipeline across two codebases, CAPC achieves 9.3x improvement vs cache-all on FastAPI. Organizational Efficiency | positive | cost/performance multiplier vs cache-all |
Reading fidelity
high
Study strength
medium
|
n=2
9.3x vs cache-all on FastAPI
|
| In the same graphify RAG pipeline, CAPC achieves 2.4x improvement vs cache-all on httpx. Organizational Efficiency | positive | cost/performance multiplier vs cache-all |
Reading fidelity
high
Study strength
medium
|
n=2
2.4x vs cache-all on httpx
|
| On the public tau-bench retail benchmark (50 tasks), CAPC is the cheapest of four strategies and attains reward exactly equal to vanilla (both 36/50, p=1.00). Organizational Efficiency | null_result | reward (task success count) and cost ranking across strategies |
Reading fidelity
high
Study strength
medium
|
n=50
36/50 tasks equal reward; p=1.00
|
| On tau-bench, query-aware compression is the most expensive strategy at +40.1% over vanilla. Organizational Efficiency | negative | cost increase relative to vanilla |
Reading fidelity
high
Study strength
medium
|
n=50
+40.1% over vanilla
|
| This work is the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark. Organizational Efficiency | positive | confirmation of a theoretical model prediction (crossover model negative-ROI) in production/public benchmark |
Reading fidelity
medium
Study strength
low
|
n=50
first production confirmation (asserted)
|