Quantized large language models can beat native small models on a triple-bottom-line edge-deployment score: Qwen3-30B and Mistral-Small in GGUF Q4 lead the 30-case comparison by balancing capability, energy, and safety; but the result depends strongly on quantization algorithm, runtime backend and evaluation scope.

Main Finding

Quantized large language models (LLMs) can be more "sustainable" for edge deployment than natively trained small language models (SLMs) when sustainability is measured jointly across capability, systems efficiency (economic), operational energy (environmental), and a simple safety proxy (social). In a 30-configuration comparison, Qwen3-30B-A3B in GGUF Q4 (llama.cpp) achieved the highest Holistic Sustainability Score (HSS = 93.38); Mistral-Small-24B / GGUF Q4 ranked second (92.40). The best SLM in the SLM-only pool was Llama-3.2-3B / BF16 (HSS 92.42), and SLMs remained competitive in the combined pool, but did not universally dominate.

Key Points

• Experimental scope: 30 measured cases — five BF16 SLMs (~3–4B params) plus five LLM families (~20–33B) each under five precision/backend configurations (BF16, INT8, NF4 4-bit, GPTQ 4-bit, GGUF Q4).
• Holistic Sustainability Score (HSS): three equally weighted pillars:
  • Economic/system (capability + latency + throughput + peak VRAM)
  • Environmental (operational energy per query and per token)
  • Social (harmful-prompt robustness measured as 1 − refusal rate, a lower attack-success-rate is better) Scores were min-max normalized within each comparison pool (so HSS is explicitly relative to the set being compared).
• Benchmarks for capability (zero-shot mean): MMLU, ARC-Challenge, HellaSwag, GSM8K, TruthfulQA.
• Safety proxy: five harmful prompts from JailbreakBench variant; responses classified by lexical refusal cues to compute attack success rate (ASR).
• Key outcomes:
  • Best overall (25-case LLM pool and combined 30-case pool): Qwen3-30B-A3B / GGUF Q4 — high throughput (153.3 tok/s), modest VRAM (19.9 GB), low energy (1.44 J/tok), HSS 93.38.
  • Mistral-Small-24B / GGUF Q4 combined strong capability (0.647) and efficiency (61.1 tok/s), HSS 92.40.
  • SLMs remain competitive: Llama-3.2-3B/BF16 top within SLM pool and placed high in combined ranking; Phi-4-mini/BF16 also ranked well in the combined pool.
• Quantization is not monotonically predictive: a lower bit-width alone does not guarantee better latency/energy. Backend, kernel support, conversion quality, and retained high-precision modules matter (e.g., GGUF/llama.cpp often improved throughput and energy; some GPTQ conversions catastrophically failed — Gemma-3-27B/GPTQ performed very poorly).
• Limitations noted by authors: small sample sizes (20 examples per task), min-max normalization sensitivity to outliers, safety proxy is limited, energy excludes PUE and grid carbon intensity, runs used different platforms/backends, and some models/checkpoints are non-aligned or reconstructed.

Data & Methods

• Models:
  • SLMs (BF16 only): Llama-3.2-3B-Instruct, Phi-4-mini-instruct, Qwen3-4B-Instruct, Gemma-3-4B-it, SmolLM3-3B.
  • LLMs: gpt-oss-20B, LLaMA-33B, Mistral-Small-24B-Instruct, Qwen3-30B-A3B, Gemma-3-27B-it.
• Quantization/backends tested per LLM: BF16 baseline, bitsandbytes INT8, bitsandbytes NF4, GPTQ 4-bit, GGUF Q4 (llama.cpp).
• Evaluation hardware: NVIDIA A100-SXM4-80GB. SLMs on Google Colab A100; LLMs on RunPod A100 pods.
• Capability: zero-shot evaluation, batch size 1, up to 20 examples per task, deterministic generation, max 80 new tokens.
• Efficiency metrics:
  • Wall-clock latency for fixed prompt set
  • Throughput (output tokens ÷ latency)
  • Peak VRAM (max of PyTorch allocation and NVML delta)
  • GPU power sampled at 50 ms; operational energy = average power × elapsed time reported per query and per token.
• Safety: five harmful prompts, refusal detected via lexical cues; ASR = 1 − refusal rate.
• HSS construction:
  • Normalize metrics within the chosen pool using min-max (higher-is-better or lower-is-better as appropriate).
  • Economic pillar = mean(Capability, Latency-neg, Throughput-pos, VRAM-neg)
  • Environmental pillar = mean(Energy per query-neg, Energy per token-neg)
  • Social pillar = ASR-neg normalized.
  • HSS = 100 × mean(Economic, Environmental, Social).
• Analysis views: SLM-only pool, full LLM pool (25 LLM cases), five within-quantization LLM pools, combined 30-case pool.

Implications for AI Economics

• Procurement and TCO: Model selection for edge deployments should use multi-criteria decision frameworks (capability, latency, memory, energy, safety) rather than single metrics (e.g., accuracy or parameter count). Quantized LLMs with optimized backends (e.g., GGUF/llama.cpp) can offer superior trade-offs that change cost and energy forecasts.
• Resource provisioning and pricing: Throughput and energy-per-token improvements from certain quantized+backend stacks materially affect operating costs and battery/runtime budgets for edge devices. Economic models (pricing, SLA guarantees) should incorporate backend-dependent performance, not just model size or nominal precision.
• Carbon & sustainability accounting: Operational energy differences observed here (e.g., multi-fold reductions using favorable quantization/backends) imply sizable differences in operational emissions. But to convert energy savings to carbon reduction, measure PUE and local grid carbon intensity and include embodied/lifecycle emissions.
• Safety and risk economics: Even small changes in refusal/ASR can alter downstream legal/operational risk and potential liability costs. Safety needs to be part of cost-benefit analyses; simplistic lexical refusal proxies are insufficient for high-stakes decisions.
• Strategy for edge AI vendors:
  • Invest in runtime/backend optimization (kernels, memory management) — these can yield outsized returns vs. purely switching model size or bit-width.
  • Use relative multi-criteria scores (like HSS) during initial screening, but follow with larger-sample, deployment-specific testing including human-in-the-loop safety evaluation.
  • Avoid blanket rules (e.g., "SLMs always best for edge"); optimal choice depends on workload, backend, and which trade-offs matter for the business case.
• Research & policy: Decision frameworks used in procurement and regulation should require reporting of capability, latency, memory, measured energy, and safety metrics together to allow sensible sustainability and cost assessments.

Caveat: HSS in this study is explicitly relative to the tested pool and uses limited safety and sample-size proxies. For economic decisions, replicate with deployment-specific workloads, runtime stacks, repeated trials and full carbon accounting before committing to a model choice.