Aggressive serving-time quantization makes small open LLMs more deterministic and less diverse: 4-bit serving of an 8B Qwen3 model narrowed recommendation choices and cut lexical diversity, while 14B and 32B models showed stylistic shifts rather than content concentration and produced no evidence of amplified stereotypes.

Main Finding

Serving-time 4-bit weight quantization (AWQ int4) can make mid-sized LLMs more deterministic in what they say (concentrating outputs onto the modal answer) without systematically amplifying stereotypical directions. This concentration effect appears at 8B in the tested Qwen3 family (fewer distinct recommendations, lower lexical diversity) but not at 14B or 32B, where quantization instead produces measurable stylistic drift. Mechanistically, token-level uncertainty rises while semantic choice diversity falls — a dissociation that makes simple logit-based audits insufficient.

Key Points

• Primary content effect (Qwen3-8B, AWQ int4 vs bf16):
  • Brand collision probability (same first-mentioned brand across two samples for the same prompt) increases by +0.051 (5.1 percentage points). Permutation test Holm-corrected p = .023 (wave-2 confirmatory).
  • Lexical diversity (type-token ratio) falls: ∆TTR = −0.0113 (p < 1e−4); MATTR-100 (length-robust) ∆ = −0.0066 (p < 1e−4).
  • Effect is heterogeneous: most prompts unchanged, a minority “collapse hard” (example: one prompt went from 4 brands in bf16 to Tesla in all int4 samples).
• Larger models (14B, 32B):
  • No significant content-concentration measures.
  • Significant stylistic drift: em-dash rate increases +0.46/1k words at 14B and +0.61/1k at 32B (Holm p ≤ .0024).
• No stereotype amplification:
  • Pre-specified tests of stereotype-direction are null across sizes and batteries; quantization amplifies determinism (concentration) but not a consistent bias toward stereotypical answers.
• Mechanistic dissociation:
  • Token-level decision entropy increases under int4 (+0.091 bits, p = .015); top-1 token probability falls (~−3.8 pp, p = .024).
  • Semantic-level measures (first-token brand distribution measured from top-50 log-probs) show greater collision (+2.6 pp, p = .023): surface-form uncertainty rises even as semantic alternatives lose mass.
• Robustness:
  • Collision effect reproduces on a full re-generation (+4.4 pp, p = .029).
  • At lower temperature (0.4) the collision contrast attenuates (direction preserved, loses significance), interpreted as ceiling effects from colder sampling.
• Limitations:
  • Single model family (Qwen3) and single quantizer per precision; recommendation battery centered on US automotive domain; English only; 32B run had different tensor-parallel setting; heterogeneous per-prompt effects.

Data & Methods

• Models and precisions:
  • Qwen3 checkpoints: 8B, 14B, 32B.
  • Weight precisions compared: W4A16 (AWQ int4), W8A16 (FP8-Marlin, weight-only), bf16.
  • Activations kept bf16; identical serving stack (vLLM 0.11.0), same hardware per precision tier within a model, consistent flags.
• Sampling and pairing:
  • Temperature 0.8, top-p 0.95, max 400 tokens.
  • For each prompt and precision tier: n = 20 samples using the same seeds across tiers to remove sampling-RNG differences.
  • Total ≈71,000 completions across the 3×3 grid and two batteries.
• Prompt batteries (leak-checked, adversarially reviewed):
  • Cultural battery: 218 country-eliciting prompts + 80 style-control prompts (with explicit “pull” labels for some prompts).
  • Recommendation battery: 96 brand-blind car-buying scenarios (brand_pull labels).
  • Gazetteers and disambiguation rules for extraction (78-country and 57-make gazetteers, adversarial tests).
• Measurements and statistical protocol:
  • Main metrics: collision probability (first-mentioned entity), rarefied entropy, type-token ratio (TTR), MATTR-100, em-dash rate, stylistic markers.
  • Mechanistic instruments: token-level top-20 log-probs per generated token; first-token brand distribution via fixed prompt “The brand I would recommend is,” top-50 probabilities (exact mapping to makes given tokenizer).
  • Analyses pre-specified in a three-wave declare-then-run protocol; unit of analysis = prompt.
  • Tests: paired sign-flip permutation tests (10,000 flips) with prompt-cluster bootstrap CIs; Holm correction within declared families.

Implications for AI Economics

• Cost vs behavioral-value tradeoffs:
  • Weight quantization reduces serving costs, but savings can change the distribution of outputs in economically meaningful ways (e.g., concentration of recommendations). Standard capability benchmarks that focus on accuracy or a single correct answer miss these distributional shifts.
  • For recommendation systems (marketplaces, retail assistants), reduced diversity can lower exposure for long-tail inventory, potentially reducing sales, discovery, and welfare even if average “accuracy” is unchanged.
• Audit and evaluation:
  • Audits should include concentration/diversity metrics (collision probability, support size across repeated samplings, mass on top-k items, lexical diversity measures) in addition to bias/stereotype checks.
  • Token-logit inspections alone may miss semantic concentration because token-level entropy can increase while semantic choices concentrate.
• Deployment mitigations and product choices:
  • If diversity of outputs matters economically, options include: using a larger model at the same precision (observed to avoid the 8B collapse here), diversity-promoting decoding strategies, explicit conditioning on available inventory or business constraints, or re-tuning/calibrating quantizers on domain-relevant calibration sets (follow-up work suggested).
  • Firms should weigh incremental serving-cost savings from aggressive quantization against potential revenue or fairness costs from reduced diversity of outcomes.
• Policy and marketplace effects:
  • Quantization-induced concentration could systematically reduce tail exposure across many deployed assistants, with broader market consequences (reduced discoverability, concentration of demand on incumbents). This is a determinism-amplification channel distinct from classical bias amplification and requires its own monitoring.
• Research/economic modeling:
  • When modeling the effect of model compression on downstream value, incorporate distributional measures (not only expected utility under a single answer) and simulate impacts on multi-option decision environments (e.g., catalog coverage, long-tail sales).

Short recommendation for practitioners: when deploying quantized LLMs for tasks where multiple valid suggestions matter, add concentration/diversity metrics to your pre-deployment audit; if you observe harmful concentration, test larger quantized checkpoints, change decoding, or condition outputs to preserve coverage.