4 cumulative citations
View corpus contextMeasuring the bits that matter: an information-theoretic metric of compressor quality predicts downstream performance and shows larger compressors are both more accurate and more token-efficient, letting modest on-device compressors pair with smaller cloud predictors to recoup near-frontier accuracy at substantially lower API cost.
An Information Theoretic Perspective on Agentic System Design
An information-theoretic estimator of mutual information between context and compressor output predicts downstream performance and shows that scaling compressors yields greater token-efficiency and accuracy gains than equivalently scaling predictors, enabling much smaller local compressors to recover near-frontier accuracy at lower API cost.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath their apparent diversity lies a recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. Despite their popularity, the design of compressor-predictor systems remains largely ad hoc, with little guidance on how compressor and predictor choices shape downstream performance. In practice, attributing gains to compression versus prediction requires costly, task-specific pairwise sweeps. We argue that these agentic system design questions are, at root, information-theoretic. Viewing the compressor LM as a noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. We show that mutual information strongly predicts downstream performance, independent of any specific task. Through an information-theoretic framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only are more accurate, but also more token-efficient, conveying more bits of information per token. A 7B Qwen-2.5 compressor, for instance, is $1.6\times$ more accurate, $4.6\times$ more concise, and conveys $5.5\times$ more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors. Applied to a Deep Research system, these principles enable local compressors as small as 3B parameters to recover $99\%$ of frontier-LM accuracy at $26\%$ of API costs.
Summary
Main Finding
Viewing compressor LMs as noisy communication channels, mutual information (MI) between the raw context X and the compressed summary Z is a task-agnostic, practical predictor of downstream performance. Scaling compressors (size and family) yields larger gains than scaling predictors: larger compressors both retain more bits about X and emit fewer tokens (higher bits-per-token), producing substantial accuracy and cost improvements. A simple Monte Carlo MI estimator that uses log-probabilities from inference servers lets practitioners predict system performance without expensive end-to-end sweeps.
Key Points
- Conceptual framing
- Compressor p(z|x) treated as a noisy channel; predictor p(y|z) consumes the channel output.
- Use I(X; Z | Q) (mutual information conditioned on a query Q) to measure how much of the original context the compressor preserves.
- Practical MI estimator (Monte Carlo)
- I(X; Z) = E_{x,z} [ log p(z|x) − log p(z) ] and approximated by ˆI ≈ (1/(N·M)) Σ_{i=1..N} Σ_{j=1..M} [ log p(z_{ij}|x_i) − log ( (1/N) Σ_{l=1..N} p(z_{ij}|x_l) ) ].
- Works with LM inference engines without full vocabulary access; clip negative estimates to zero in practice.
- Rate and distortion
- Define rate R = I(X;Z|Q) / L (bits per output token) and distortion D = 1 − ACC(Z).
- Bits-per-token (information rate) strongly correlates with downstream accuracy and perplexity (reported r = −0.84, R² = 0.71).
- Empirical scaling results (across five datasets: LongHealth, FinanceBench, QASPER, WildChat, FineWeb)
- Larger compressors increase MI and accuracy while producing shorter compressions (more token-efficient).
- Example (abstract / scaling highlights): Qwen-2.5 7B vs 1.5B compressor — ~1.6× more accurate, ~4.6× more concise, and ~5.5× more bits of MI per token.
- Compressor FLOPs-per-generation scale sublinearly with model size because larger compressors emit fewer tokens.
- Scaling compressors produces larger downstream improvements than scaling predictors: e.g., on LongHealth scaling compressor from 1B→7B gave +60% accuracy while scaling predictor 70B→405B gave only +12%.
- Meta-analysis hierarchy of importance: compressor model family > compressor size > predictor size.
- Practical system benefit
- In a Deep Research-style pipeline, local compressors as small as 3B recovered ~99% of frontier-LM accuracy while reducing API costs to ~26% (≈74% cost reduction).
Data & Methods
- Datasets
- LongHealth (synthetic clinical reports), FinanceBench (10-K filings), QASPER (scientific QA), WildChat (conversational), FineWeb (web pages).
- Models
- Compressors: open families Llama-3, Qwen-2.5, Gemma-3 (sizes 1–12B; some experiments on Qwen-3 MoE).
- Predictors: GPT-4o and Llama-3 family (1B–405B), Qwen-2.5 72B.
- Evaluation
- QA accuracy measured (LongHealth, FinanceBench, QASPER) with a GPT-4o-mini judge; perplexity used for WildChat and FineWeb (Llama-3.1-8B).
- Experiments run with S = 5 random seeds.
- MI estimation details
- Monte Carlo estimator using p(z|x) scores from inference servers; proxy models sometimes used for log-probabilities when compressor models are small/miscalibrated.
- Negative Monte Carlo artifacts clipped to zero.
- Compute accounting
- FLOPs-per-generation estimated from model size and generated token counts to compare cost trade-offs between compressors and predictors.
- Additional analyses
- Rate-distortion curves, prompt conciseness ablations (3/6/9 sentences), error categorization of compressor failures.
Implications for AI Economics
- Cost and deployment strategy
- "Front-loading" compute into local compressors (on-device or edge) can substantially reduce cloud predictor/API usage and cost, because compressors can compress more effectively and emit fewer tokens as they scale.
- Sublinear FLOPs-per-generation with compressor scaling improves cost-effectiveness of larger local compressors.
- Example business impact: running 3B local compressors to recover ~99% of frontier accuracy while cutting API spend by ~74% (to ~26% of baseline) materially lowers operating costs for multi-LM services.
- Product design and pricing
- Providers and integrators should prioritize investing engineering/compute budget into compressor selection and tuning (model family and size) before scaling predictors.
- Token-efficiency gains reduce per-request billing (token-based pricing), so compressor improvements translate directly into lower variable costs.
- As more capable compressors run on-device, demand for constant large-cloud-predictor compute may decline, pressuring cloud model monetization; hybrid pricing or managed-compression offerings may emerge.
- Experimentation and decision-making efficiency
- The MI estimator provides a task-agnostic proxy to evaluate compressor quality, enabling cheaper, faster ablation/swap of components without full end-to-end sweeps—lowering experimentation cost and time-to-deploy.
- Risks and caveats (economic relevance)
- MI estimator is approximate and requires access to p(z|x) scores (or proxies); small compressors can miscalibrate log-probs, requiring proxy evaluations.
- Dataset-dependent saturation: on some corpora MI saturates at modest sizes, so additional compressor scaling yields diminishing returns; economic optimization must consider dataset characteristics.
- Privacy, regulatory, and hardware costs: moving computation on-device reduces recurring cloud costs but raises upfront device-capability and model-distribution concerns (security, updates, monetization).
- Market effects: reductions in cloud compute demand for prediction could shift pricing, competitive dynamics, and business models for model-hosting providers.
Summary takeaway: invest first in compressor model family/size and measure bits-per-token (MI rate) as a cheap, predictive signal of downstream utility—this can substantially lower token and API costs while preserving accuracy, changing where value and costs accrue in multi-LM product architectures.
Assessment
Paper Typecorrelational
Evidence Strengthmedium — The paper reports consistent, cross-dataset and cross-model correlations between the mutual-information estimator and downstream performance and compares compressors and predictors across sizes; this provides robust empirical support for the claim that mutual information is a task-independent predictor. However the work is observational/experimental rather than causal, depends on the accuracy and assumptions of the mutual-information estimator, and may be sensitive to dataset selection, prompt engineering, and hyperparameter choices not exhaustively explored.
Methods Rigormedium — Strengths include an information-theoretic framing, a concrete estimator, evaluations across five datasets and three model families, and concrete cost/efficiency comparisons (including a Deep Research application). Weaknesses include potential biases or estimation error in the mutual-information estimator, limited transparency about estimator/hyperparameter robustness in diverse settings, and absence of randomized interventions that would more cleanly separate compressor versus predictor causal effects.
SampleEmpirical experiments across five datasets (task types not exhaustively listed), three LM families (including Qwen-2.5), and multiple compressor sizes (examples: 1.5B, 3B, 7B) paired with predictors of varying sizes; evaluation includes downstream task accuracy, token counts, an estimated mutual-information metric, and an applied Deep Research system cost/accuracy comparison.
Themesadoption productivity innovation
IdentificationFrame the compressor LM as a noisy channel and estimate the mutual information between original context and its compressed text using a proposed estimator; show empirically across compressor/predictor pairs, datasets, and model families that this mutual information strongly predicts downstream task performance and token-efficiency.
GeneralizabilityEvaluations limited to three model families (may not hold for other architectures or emerging LMs), Five datasets may not represent the full diversity of real-world tasks, domains, or languages, Mutual-information estimator may rely on distributional assumptions and could be biased for other data regimes or compression formats, Results pertain to text-based compressor outputs; non-text compressions or multimodal contexts are untested, Cost savings and latency claims depend on current API pricing and deployment details which can change
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| We introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. Other | positive | compression quality measured via mutual information between context and compression |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Mutual information strongly predicts downstream performance, independent of any specific task. Output Quality | positive | downstream performance (task accuracy / task performance) |
Reading fidelity
high
Study strength
medium
|
n=5
|
| Larger compressors are more accurate and more token-efficient, conveying more bits of information per token. Output Quality | positive | accuracy, token-efficiency, mutual information per token |
Reading fidelity
high
Study strength
medium
|
not reported
|
| A 7B Qwen-2.5 compressor is 1.6× more accurate, 4.6× more concise, and conveys 5.5× more bits of mutual information per token than its 1.5B sibling. Output Quality | positive | accuracy; conciseness (tokens); mutual information per token |
Reading fidelity
high
Study strength
high
|
1.6× more accurate, 4.6× more concise, 5.5× more bits per token
|
| Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors. Output Quality | positive | downstream performance as a function of scaling compressor vs predictor |
Reading fidelity
medium
Study strength
medium
|
n=5
|
| Applied to a Deep Research system, local compressors as small as 3B parameters can recover 99% of frontier-LM accuracy. Output Quality | positive | accuracy recovery relative to frontier LM |
Reading fidelity
high
Study strength
medium
|
99%
|
| Applied to a Deep Research system, these local compressors can achieve this while incurring 26% of API costs. Firm Productivity | positive | API costs (as a fraction of baseline) |
Reading fidelity
high
Study strength
medium
|
26%
|
| Design of compressor–predictor systems has been largely ad hoc with little guidance on how compressor and predictor choices shape downstream performance. Other | negative | availability of principled design guidance for compressor–predictor systems |
Reading fidelity
high
Study strength
speculative
|
not reported
|