Plain-text prompts can make LLMs 'overflow', producing heavy-tailed output that raises serving costs and energy use; BenchOverflow documents this across nine models and nine prompting strategies and finds a short conciseness reminder markedly reduces excessive token generation.

Main Finding

Plain-text, non-adversarial prompts can reliably induce excessive token generation in modern LLMs — a phenomenon the authors call "Overflow." Using BenchOverflow (a benchmark of nine plain-text prompting strategies) and a fixed generation budget of 5,000 new tokens, the study shows pronounced rightward shifts and heavy tails in output-length distributions across both open- and closed-source models. A simple, model-agnostic mitigation (a fixed conciseness reminder prepended to prompts) consistently reduces tail risk and cap-saturation rates for most strategies and models.

Key Points

• Definition: Overflow = prompt-induced excessive text generation measured in absolute token counts (native tokenizer).
• Attack surface: Overflow arises from ordinary, benign-seeming prompts (no jailbreaks, adversarial suffixes, or corpus/control privileges required). Example vectors include explicit forced length, implicit large enumeration, infinite generation requests, recursive expansion, quoting large public-domain texts, tokenizer-sensitive inputs, etc.
• Benchmark (BenchOverflow): Nine prompting strategies, each instantiated with >300 prompts (systematically generated via a meta-prompting procedure using GPT-4o with positive/negative examples).
• Models evaluated: 9 representative systems (6 open-source: Qwen3 variants, LLaMA-3 variants, Gemma models; 3 closed-source: GPT-5, Gemini-2.5-Flash, Claude-Sonnet).
• Experimental protocol: single system message ("You are a helpful assistant.") + single user prompt; native tokenizers; fixed max_new_tokens budget M = 5,000; open-source sampling: do_sample=True, temperature=1.0; closed-source: provider defaults. For each model/prompt: 4 runs; for each model ~4,000 total generations (BenchOverflow + benign baseline OASST2).
• Metrics: distributional analysis (histograms, ECDFs), cap-saturation rates CSR@{1k,3k,5k} (frequency of generations exceeding thresholds), within-prompt variance, cross-model/prompt correlation.
• Results summary: Overflow is widespread but heterogeneous: many prompts push models toward or up to the 5k token budget, giving heavy right tails and nontrivial CSR values; behavior is reproducible across runs and models but varies by model family and prompting strategy.
• Lightweight mitigation: a generic conciseness reminder prepended to the user prompt reduces right-tail mass and lowers CSR across most strategies and models without requiring model internals or bespoke defenses.

Data & Methods

• Prompt generation:
  • Meta-prompt template that encodes attack vector description, required/prohibited components, positive examples (5–8), negative contrasts (3–4), and JSON output schema.
  • GPT-4o used to generate prompts under default decoding (temperature=1.0).
  • Nine strategy categories: change forms, explicit forced length, implicit large enumeration, infinite generation, recursive details, roleplay simulation, tokenizer stress, quote, stepwise explanation.
• Dataset and sampling:
  • BenchOverflow: >300 prompts per strategy (diverse, statistically robust).
  • Benign baseline: OpenAssistant Conversations (OASST2). Deterministic sampling of 100 prompts per dataset to compare against overflow-oriented prompts.
• Execution:
  • 4 independent runs per prompt–model pair.
  • Native token counts recorded from model providers or model tokenizer.
  • Uniform generation budget M = 5,000 new tokens to ensure comparability.
• Evaluation metrics:
  • ECDFs and histograms to capture distributional shifts and tails.
  • CSR@1k/3k/5k to quantify frequency of near-budget or saturated generations.
  • Within-prompt variance and per-prompt cross-model correlation to assess reproducibility and heterogeneity.
• Limitations noted by authors:
  • The prompt generation process used GPT-4o and thus could reflect generator biases.
  • Closed-source provider defaults (sampling, penalties, safety filters) are not controlled and can influence outputs.
  • Tokenization differences across models mean absolute token counts are model-specific; CSR thresholds are therefore empirical, not universal.

Implications for AI Economics

• Direct operating cost impact:
  • Per-token billing models make Overflow an immediate cost driver: heavy-tail generations multiply per-request spend. Even non-adversarial users can produce outsized bills if prompts elicit long completions.
  • At scale, many small overstretches aggregate into large, predictable increases in cloud/API spend and capacity requirements.
• Throughput, latency, and SLA risk:
  • Long generations increase latency and occupy inference slots longer, reducing throughput and increasing queuing for other tenants or users. This creates systemic tail-risk for service availability and SLAs (e.g., higher p95/p99 latency).
  • "Denial-of-Wallet" and compute-amplification attacks are practical without jailbreaks: attackers or careless users can leverage Overflow to exhaust rate limits, budgets, or shared capacity.
• Capacity planning and procurement:
  • BenchOverflow provides operationalizable metrics (ECDFs, CSR@τ) that should be integrated into procurement and vendor-evaluation processes. Buyers should evaluate models not only for accuracy or alignment but for length-control robustness and tail risk.
  • Selection of models for cost-sensitive deployments should include stress tests like BenchOverflow to estimate expected and worst-case token consumption.
• Environmental externalities:
  • Excess tokens imply more compute and energy; scaled across many requests, this increases carbon footprint and should be considered in sustainability and carbon-accounting models.
• Policy and pricing design:
  • Providers and platform operators may need to design pricing and throttling to internalize overflow risk — e.g., progressive pricing for unusually long completions, stricter per-request token caps, or token-budget alerts.
  • Economically efficient defenses include lightweight user-level mitigations (conciseness reminders), enforced max_tokens, per-user quotas, tail-aware throttling, and monitoring/alerting on CSR metrics.
• Risk management:
  • Overflow creates an operational tail-risk that warrants inclusion in risk registers for production deployments (financial institutions, customer support, multi-tenant APIs).
  • Firms should model both expected token spend and tail scenarios (e.g., probability mass in CSR@3k/5k) when forecasting costs and provisioning capacity.
• Design and model training:
  • Providers should consider length-control mechanisms during model training and evaluation (reward concision when appropriate, stricter termination priors), since ad hoc runtime truncation can degrade response quality.
  • BenchOverflow can be used to evaluate whether defenses reduce overflow without degrading utility — relevant for cost-performance trade-offs.

Practical recommendations for practitioners and procurement teams - Include CSR@τ (e.g., τ = 1k, 3k, 5k) and ECDF tail analyses in vendor evaluations. - Run BenchOverflow-like stress tests as part of QA to quantify tail token consumption. - Implement light-weight prompt-level mitigations (conciseness reminders) and hard caps or per-user quotas to limit financial and latency exposure. - Monitor per-user and per-endpoint token distributions; set alerts for abnormal tail activity. - Factor overflow-driven tail risk into cost forecasts, SLA design, and carbon-accounting.