A lightweight multimodal model can predict when large LLMs 'over-think' and pick cheaper per-query reasoning budgets, cutting inference costs by up to 99% in tests while generally maintaining or improving accuracy across document tasks.

Main Finding

A lightweight (~1B-parameter) multimodal model (DRB) can pre-flight estimate how well much larger frontier LLMs will perform on document tasks at different reasoning budgets. These estimates are accurate enough to (a) avoid the widespread “over‑thinking” penalty, (b) select lower-cost per-sample reasoning budgets, and (c) produce large cost savings while often preserving or improving accuracy. On the paper’s BudgetDoc benchmark, DRB attains a weighted F1 = 0.753 and class‑6 recall = 0.932; using DRB to pick budgets reduced cost in 13/15 model–dataset configurations and matched or improved F1 in 9/15.

Key Points

• Over-thinking penalty: Increasing test‑time reasoning tokens is not always helpful and can reduce accuracy or waste money; the optimal budget is task-, model-, and sample-specific.
• In-flight self-assessment is insufficient: Provider-side dynamic-thinking controls (e.g., Gemini auto-thinking, GPT reasoning effort) and models’ own self-predictions have weak correlation with realized token use and cost, producing “pricing reversals.”
• Pre-flight estimator approach: A small external estimator that reads the document and predicts model performance at candidate budgets can impose hard budget caps before expensive inference.
• BudgetDoc dataset: First supervised benchmark for per-sample model×budget performance estimation — exhaustive profiling of 360 base documents × 5 models × 5 budgets = 9,000 labeled configuration→outcome pairs; held‑out test set = 100 documents → 2,500 test pairs.
• DRB model: ~1B parameters composed of SigLIP‑2 visual encoder (428M), Qwen3‑0.6B text encoder, a cross‑page fusion transformer, learned embeddings for (model, budget), and an MLP head to predict a 7‑class ordinal performance label (classes derived from discretized F1 bins).
• Estimation performance: weighted F1 = 0.753 on BudgetDoc test; highest reliability on Class 6 (near‑perfect outputs) with recall = 0.932. Intermediate classes are harder (F1 0.46–0.74), reflecting stochastic partial reasoning behavior.
• Deployment method: hierarchical scanning — DRB predicts performance from lowest to highest budgets and stops early (e.g., when class 6 is predicted); selection ties broken by lower cost. Estimator overhead is accounted for in adjusted cost calculations.
• Downstream results (held-out documents): DRB-guided selection reduced costs by 5.2%–99% depending on model/dataset; largest savings on expensive GPT models. Accuracy was maintained or improved in the majority of configurations (9/15), confirming over‑allocation harms in practice.

Data & Methods

• Tasks and sources:
  • RVL‑CDIP: modified multi‑page classification (three concatenated PDFs per example).
  • TAT‑DQA (TAT‑QA): hybrid numeric/tabular reasoning over reports.
  • CheckboxQA: scanned form checkbox extraction.
• Dataset construction:
  • 120 unique base samples per task → 360 unique base contexts.
  • Exhaustive evaluation across five frontier models (gemini-2.5-flash, gemini-3-flash-preview, gemini-3.1-flash-lite-preview, gpt-5.1, gpt-5.2) and five budgets b ∈ {0, 512, 1024, 1536, 2048}, yielding 9,000 labels.
  • Performance discretized into seven ordinal classes (0..6) from failure to near‑perfect.
  • Split by unique documents: 240 base contexts → training (6,000 pairs), 20 → validation (500 pairs), 100 → test (2,500 pairs).
• DRB architecture:
  • Visual: SigLIP‑2‑Large‑patch16‑512 (428M) to get page embeddings; 12‑layer decoder‑style Llama fusion transformer for cross‑page attention and pooling.
  • Text: Qwen3‑0.6B for prompt encoding; linear projection to align modalities.
  • Conditioning: learned continuous embeddings for model identity and budget.
  • Fusion: concatenation of z_doc, z_prompt, e_budget, e_model → 2048‑unit MLP → softmax over 7 classes.
  • Loss: cross‑entropy over classes.
• Evaluation & cost accounting:
  • Primary metric: task F1. Also report per‑sample API USD cost and latency.
  • Adjusted cost includes estimator infrastructure overhead (empirical scanning latency × hardware cost assumption: $0.00015/s on a T4 equivalent).
• Downstream procedure:
  • Hierarchical scanning of budget classes using DRB predictions to choose the minimal sufficient budget; exploratory model‑selection experiments also performed.

Implications for AI Economics

• Substantial direct cost savings: Per-sample pre-flight estimation enables selecting much lower reasoning budgets without sacrificing accuracy, yielding dramatic reductions in billed compute (up to ~99% in some configurations). For expensive frontier models, per-query savings can be economically transformative for high-volume applications.
• Better price‑performance alignment: External estimators reduce the inefficiency from provider-side self‑managed reasoning (dynamic auto-thinking) and can mitigate pricing reversals and per-query token variance that erode nominal cost advantages.
• New demand for estimation services and tooling: There is a market opportunity for lightweight performance‑estimators (or “pre-flight” optimization layers) as value-added services that sit in front of provider APIs, potentially offered by third parties or integrated by platform/builders.
• Incentives and provider responses:
  • Providers may adapt pricing or expose finer-grained, more predictable metering to reduce arbitrage and make in-flight costs more stable.
  • Providers might offer dedicated estimator endpoints, configurable hard caps, or explicit per‑request cost guarantees in response to third‑party optimization layers.
• Effects on model selection and market competition: Accurate cross‑model pre-flight estimation can facilitate per-query model routing or cascades (choosing the cheapest model that will meet quality thresholds). That can intensify competition on predictable price‑performance rather than headline model capability alone.
• Operational trade-offs and risks:
  • Estimator cost must remain small relative to savings; this paper accounts for estimator overhead but real-world deployment must monitor drift as target models change.
  • An arms race is possible: providers might change internal behavior (e.g., token accounting, dynamic thinking internals) to reduce predictability or to capture more billing; continuous retraining and multi‑provider estimator training will be needed.
  • Potential for gaming: estimating services could be exploited to fingerprint providers or infer internal behaviors; privacy and contractual issues may arise.
• Policy and procurement: Organizations negotiating large API volumes could mandate or prefer provider features that enable predictable, capped inference costs; estimators can inform procurement decisions and SLAs.

Limitations and future directions (economic relevance) - Generalization: DRB was trained on a fixed set of providers and tasks; broader provider coverage and continual retraining are needed for robust, production‑grade economic impact. - Intermediate-class uncertainty: imperfect estimates in middle difficulty cases limit fully riskless cost cutting; decision-makers may need utility-weighted thresholds (cost vs. expected quality loss). - Wider scope: extending to conversational, multi-turn, or agentic workloads (where token variance is larger) could yield even greater economic returns but will require richer datasets and frequent re-profiling. - Market design: findings motivate new billing designs (e.g., guaranteed max-thinking caps, discounted pre-flight guarantees, or outcome‑based pricing) that could align provider incentives with efficient per-sample compute allocation.

Short takeaway: Pre-flight, small multimodal estimators like DRB are a practical lever to make LLM inference compute‑optimal at scale, with strong implications for cost structures, supplier competition, and the design of provider pricing and control features in the AI economy.