There is no one-size-fits-all compression for edge AI: quantization typically yields predictable size and latency wins, but structured pruning can both erode task performance and — by breaking format alignments — inflate deployed artifacts and increase latency. Edge deployments must pick compression techniques by task, model family and hardware, or risk surprising slowdowns and hidden accuracy failures.

Main Finding

No single compression technique reliably dominates for edge deployment — the best choice depends on the task, the model family, and the target hardware. Quantization often gives the largest practical wins for on-device language understanding (QA) by shrinking storage and power cost without changing topology, while structured pruning can deliver large model-size and MAC reductions that benefit image segmentation. However, pruning can also harm accuracy sharply (pruning‑induced forgetfulness), break deployment-format alignment (inflating artifact size and per-token decode cost), and increase on-device latency. Recovery methods (e.g., LoRA) can mask failures (high strict accuracy but severe prediction skew). Device-level behavior must be measured end-to-end (prefill vs decode) and analyzed hardware‑aware; otherwise nominal compression numbers mislead deployment economics.

Key Points

• Empirical, deployment-focused pairing: survey of ~20+ works that report real-hardware measurements plus an original cross-task deployment study (LLMs and segmentation models) on GPU, CPU, and Raspberry Pi.
• No universal winner:
  • Question answering (LLMs): GGUF quantization (Q5_K_M) of Qwen3.5 0.8B reached 93.85 SQuAD F1 and 92 EM. At the same nominal precision, structured pruning (1% MLP channels) lost ~16 F1.
  • Segmentation: pruning reduced model size by nearly 80% with near-constant mIoU; quantization alone did not change parameter count or MACs (so less benefit for topology‑heavy segmentation speed).
• Pruning-induced forgetfulness: even small structured pruning ratios (e.g., 1% of MLP channels) can cause substantial downstream quality loss across evaluated LLM families; LoRA fine-tuning after pruning recovers only part of the loss.
• Neural-flow graph analysis identifies where pruning damage concentrates inside a model family:
  • For Qwen3.5, 77% of measured distortion localizes to the gate projection.
  • For Qwen2.5, 67% concentrates on the down projection. This layer-locality explains family‑specific vulnerability and guides targeted recovery.
• Two-stage inference decomposition (prefill vs decode):
  • Prefill: compute-bound.
  • Decode: memory-bandwidth-bound. Structured pruning increased latency in every matched baseline-vs-pruned pair, because pruned models generated more tokens (up to 2.7× for Gemma) and, when pruning breaks k-quant super-block alignment, the deployable weight set can grow by 21–49% — raising per-token cost and Raspberry Pi latency up to 3.4×.
• Compression can mask failure modes (prior-collapse): a LoRA-recovered variant remained syntactically parseable and achieved 71% strict BoolQ accuracy while producing 97/100 predictions for a single class; balanced accuracy was only 52.6% — illustrating why reporting strict accuracy alone is misleading.
• Practical artifact issues: pruning can increase the stored/deployable artifact size (format-dependent) by destroying quant-block alignment; quantization can be limited by missing device kernels (e.g., INT4 often unsupported on microcontrollers).
• Open-sourced artifacts and code: authors provide the experiment code and artifacts.

Data & Methods

• Models and tasks:
  • LLM families: three ~1B-parameter families (e.g., Qwen2.5, Qwen3.5, Gemma/TinyLlama variants).
  • NLP tasks: SQuAD (QA, reported F1 and EM), BoolQ and Natural Questions.
  • Vision tasks: six segmentation models (DeepLabV3+, SegFormer, Mask2Former, SAM derivatives) evaluated with mIoU and mean accuracy.
• Compression techniques evaluated:
  • Structured pruning (channel / projection pruning).
  • Post‑training quantization (GGUF Q5_K_M and other PTQ variants).
  • LoRA recovery / low-rank fine-tuning after pruning.
  • Knowledge distillation (for segmentation experiments), and combinations (pruning + quant).
• Hardware targets:
  • GPU, CPU, and Raspberry Pi (and comparisons to Jetson platforms in the survey context).
• Measurements and analyses:
  • Measured end-to-end latency, artifact (checkpoint) size, MACs/parameter counts, power/energy where available.
  • Performed 18 matched baseline-vs-pruned run pairs to isolate pruning effects.
  • Decomposed inference into prefill (compute-bound) and decode (memory-bandwidth-bound) phases; measured token counts and per-token costs.
  • Neural-flow graph analysis: first-order, layer-additive measure of local signal distortion introduced by pruning, used to rank damage and locate concentrations of distortion.
  • Reported both strict accuracy and balanced accuracy + prediction skew to expose collapse modes.
• Representative quantitative findings:
  • Qwen3.5 0.8B + Q5_K_M (GGUF) quant: 93.85 SQuAD F1 and 92 EM.
  • Structured pruning (1% MLP channels) at same precision: ≈16 F1 drop on SQuAD for some trajectories.
  • Pruning can inflate quant-deployable weights by 21–49% due to broken k-quant block alignment, increasing Raspberry Pi latency up to 3.4×.
  • Pruned models generated more tokens (up to 2.7× increase observed for Gemma).
  • LoRA-recovered model: 71% strict BoolQ accuracy yet 97/100 predictions to a single class; balanced accuracy 52.6%.
  • In all 18 matched pruned vs baseline pairs, structured pruning raised per-token decode cost.

Implications for AI Economics

• Procurement and deployment decisions must be hardware- and task-aware:
  • Don’t choose compression based only on parameter count or nominal sparsity/bit-width. Quantization can deliver storage and energy savings cheaply for LLM QA workloads, but pruning may be necessary when reducing MACs matters (e.g., segmentation) — and pruning can carry hidden costs.
• Hidden costs and operational risk:
  • Accuracy loss (pruning-induced forgetfulness) reduces model utility and may require re‑training or human oversight; these are real economic costs (retraining compute, downtime, quality loss).
  • Artifact inflation (format misalignment) can force upgrades to more expensive devices or cloud fallbacks, negating compression savings.
  • Failure modes that keep high strict accuracy while collapsing predictions (skew) can mislead monitoring and lead to bad business decisions; balanced metrics and skew reporting should be required in SLAs.
• Infrastructure investment trade-offs:
  • Two-stage inference split implies different optimization targets: investment in compute acceleration (e.g., better CPU/GPU) reduces prefill cost; investment in memory bandwidth or quant-format support reduces decode cost. For LLMs, decode-phase bandwidth is often the bottleneck on edge devices.
  • Device selection should consider supported quant kernels (INT8, INT4) and format block sizes (k-quant alignment) — choosing devices and runtimes that avoid alignment pathologies can yield material cost savings.
• Design and research priorities with economic impact:
  • Favor hardware-aware compression pipelines (pruning+quantization+format-aware packing) to capture both storage and runtime gains.
  • Require deployment benchmarks measured end-to-end on target hardware (latency, energy, artifact size, balanced accuracy, skew) as part of procurement evaluation.
  • Invest in lightweight recovery/fine-tuning methods that target layer-local distortion (guided by neural-flow analyses) to reduce retraining costs.
• Operational recommendations:
  • Report both strict and balanced accuracy plus prediction skew in deployment metrics and SLAs.
  • Benchmark candidate compressed artifacts on the exact target device and runtime, including end-to-end serving logs to capture prefill/decode splits and token-generation differences.
  • Prefer quantization for QA/LLM prompt-encoding workloads where supported; prefer pruning and distillation for segmentation when MAC reduction yields runtime wins — but test for alignment and artifact growth.
• Broader economic effect:
  • Accurate, device-specific measurement reduces costly surprises (unexpected latency, energy, or cloud fallbacks). The paper’s open-sourced artifacts lower replication and evaluation cost, enabling more transparent, economically prudent edge deployment decisions.

Source: Das et al., "Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment" (Ohio State University; arXiv preprint; code: github.com/Arnavvvkumar/deployment).