A single backward pass lets WANDA focus pruning budget on loss-sensitive neurons: F-WANDA raises knowledge-task accuracy by ~1.5 percentage points at 50% sparsity while cutting pruning energy roughly threefold versus the costly SPARSEGPT method.

Main Finding

F‑WANDA is a drop‑in modification of WANDA that uses a single backward pass over the same calibration corpus to compute an empirical Fisher scalar per output neuron and then reallocates the per‑row keep budget proportionally (via a water‑filling allocator). At 50% unstructured sparsity on LLAMA‑2 models, F‑WANDA matches WANDA on fluency (WikiText‑2 PPL) while improving knowledge performance (5‑shot MMLU) by ≈1.4–1.6 percentage points and sits on the Pareto frontier of quality vs. pruning energy: it delivers higher quality than WANDA and is substantially cheaper than SPARSEGPT.

Key Points

• Core idea: compute row‑aggregated empirical Fisher ω¯i = (1/N) Σt (∂Lt/∂y t,i)^2 from one backward pass, use √ω¯i to allocate each row’s keep count ki (instead of uniform per‑row budget), then keep the top ki weights per row by WANDA’s |Wij|·∥Xj∥2 score.
• Degeneracy avoided: multiplying WANDA scores by a per‑row scalar does not change within‑row ranking; F‑WANDA uses Fisher to change the per‑row budget (ki), not to rescale weights.
• Budget allocation: global keep budget K = ⌈(1−s) dout·din⌉; ki ∝ √ω¯i with clamps [1, din], implemented via iterative water‑filling and largest‑remainder rounding to ensure integer sum K.
• No weight updates, no extra calibration data, only one extra backward pass (no Hessian inversions).
• Primary empirical gains at 50% unstructured sparsity:
  • LLAMA‑2‑7B: WikiText‑2 PPL 6.85 (F‑WANDA) vs 6.92 (WANDA); 5‑shot MMLU 43.1% vs 41.5% (+1.6 pp).
  • LLAMA‑2‑13B: PPL 5.93 (F‑WANDA) vs 5.97 (WANDA); 5‑shot MMLU 52.4% vs 51.0% (+1.4 pp).
• Efficiency/sustainability:
  • F‑WANDA pruning energy (one shot): 4.3 kJ (7B) and 10.1 kJ (13B).
  • Compare SPARSEGPT: 12.6 kJ (7B) and 27.0 kJ (13B) — F‑WANDA ≈2.6–2.9× lower pruning energy than SPARSEGPT.
  • Prune time / peak GPU memory (single H100 80GB): WANDA 5 min / 14 GB (7B); F‑WANDA 12 min / 30 GB (7B); SPARSEGPT 35 min / 22 GB (7B). (13B numbers: WANDA 10 min/26GB; F‑WANDA 28 min/58GB; SPARSEGPT 75 min/40GB.)
  • Inference energy per token is the same across pruning methods at matched sparsity (savings realized at compression time).
• Robustness & ablations:
  • Calibration size: performance plateaus ≥128 sequences of 2048 tokens (the default).
  • Using (E[g])^2 instead of E[g^2] collapses allocation to uniform; sampling labels (true Fisher) similar to empirical Fisher.
  • Largest mask differences vs WANDA occur in mid‑network MLP down‑projection layers (factual stores).
• Limitations:
  • No benefit under strict hardware N:M constraints (e.g., 2:4) because per‑block keep counts are fixed; reduces to WANDA there.
  • Extra backward pass increases peak memory (could require model parallelism for very large models).
  • Evaluated on LLAMA‑2 family and C4 calibration shard only; cross‑family and cross‑corpus robustness untested.

Data & Methods

• Models: LLAMA‑2‑7B and LLAMA‑2‑13B (official HF checkpoints, BF16).
• Calibration: 128 sequences × 2048 tokens sampled from a fixed C4 shard (same as WANDA baseline).
• Pruning targets: 50% unstructured sparsity; also reported 2:4 (where F‑WANDA reduces to WANDA).
• Saliency baseline (WANDA): Sij = |Wij| · ∥Xj∥2 (one forward pass).
• F‑WANDA additions:
  • One backward pass to collect per‑row empirical Fisher ω¯i = (1/N) Σt (∂Lt/∂y t,i)^2.
  • Compute vi = √ω¯i and allocate integer keep counts ki by iterative water‑filling + largest‑remainder rounding so Σi ki = K.
  • For each row, keep top ki entries by WANDA score.
• Complexity: O(N·|θ|) (forward+backward over calibration) vs SPARSEGPT’s layerwise O(din^3) Hessian inversions; practical pruning wall‑clock and energy substantially lower than SPARSEGPT.
• Evaluation: WikiText‑2 perplexity, 5‑shot MMLU, seven‑task zero‑shot suite; energy measured by NVML (nvidia‑smi) sampled at 10 Hz with idle subtraction, five repeats, 95% CIs reported in appendix.

Implications for AI Economics

• Low marginal cost of higher quality: F‑WANDA demonstrates that a small additional upfront compression cost (one backward pass) can meaningfully preserve model knowledge at fixed sparsity. This improves the cost‑effectiveness of one‑shot pruning strategies used in deployment pipelines.
• Pruning energy matters: SPARSEGPT produces high quality but at much higher pruning energy and wall‑clock time. For organizations that repeatedly compress or re‑compress models (e.g., after fine‑tuning), the lower pruning energy and faster turnaround of F‑WANDA reduce operational costs and carbon footprint associated with compression workflows.
• Fleet‑scale impact: reported one‑shot pruning costs are small in absolute terms (e.g., 10.1 kJ for a 13B prune), but at fleet scale or at frequent re‑compression cadence these multiply; replacing expensive schemes with F‑WANDA can cut that portion of operational energy by ≈2.6–2.9× while improving (not degrading) knowledge retention.
• Inference vs compression trade: inference energy per token is determined by the deployed sparsity/density and hardware support; choice of pruning algorithm affects compression energy and quality but not per‑token inference energy at a given sparsity. Therefore, algorithm choice is an operational cost decision (pruning-time energy, wall‑clock, memory) with direct quality consequences that influence downstream utility and possibly the need for re‑serving denser models (which would increase inference costs).
• Practical deployment note: benefits accrue most for unstructured sparsity regimes or when hardware/software supports using unstructured masks efficiently; under strict hardware N:M constraints F‑WANDA gives no advantage over WANDA, so economic gains depend on target deployment format.
• Recommendation for economic evaluation: reporting pruning-time energy and latency alongside inference costs is important for meaningful cost–quality comparisons of compression methods; F‑WANDA provides a pragmatic middle ground for organizations balancing compression energy budgets against knowledge retention in LLMs.

If you want, I can produce a short table comparing the key numeric cost/quality tradeoffs (prune energy, time, MMLU gains) for quick reference.