Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference

Vision-Language Models (VLMs) are the perceptual backbone of embodied AI, but their energy footprint on edge hardware remains poorly understood. Existing efficiency efforts focus predominantly on reducing visual tokens, implicitly treating visual processing as the dominant energy cost. We overturn this implicit assumption through the first systematic energy profiling of on-device VLM inference, spanning five models across three architecture families, four input resolutions, and two hardware platforms (NVIDIA RTX 3070 and Jetson Orin NX). Our analysis yields three findings. First, average inference power is a model-intrinsic constant, invariant to input resolution, image complexity, and prompt type, with less than 5% variation across all conditions. This means that all energy variation across inputs must arise from variation in inference time, not from variation in power draw. Second, each output token costs 11 to 39x more wall-clock time than each input token due to the compute-bound and memory-bound asymmetry between prefill and decode, making output token count the dominant driver of both latency and energy. Third, image complexity, measured by the number of objects in an image, induces up to 4.1x energy differences at identical resolution. This variation arises not from increased visual processing cost, but from differences in output length. These findings expose a fundamental limitation of visual token pruning: even removing all visual tokens saves at most 10% of total energy for fixed-token models. Across models spanning 1 billion to 8 billion parameters, controlling output length saves up to 97% of total energy, with the energy dominance of decoding growing stronger at larger model scale. In short, the true energy bottleneck in edge VLM inference is not what the model sees, but how much it says.

Summary

Main Finding

The dominant energy cost for on-device Vision-Language Model (VLM) inference is decoding (generation), not vision processing. Average power draw is essentially a model-specific constant (a “power fingerprint”), so energy variation across images/queries comes almost entirely from differences in runtime — and runtime is overwhelmingly driven by the number of output tokens. In practice, decoding consumes 86–97% of total inference energy, and each output token costs 11–39× more wall-clock time (and thus energy) than each input token.

Key Points

Power fingerprint
- Per-inference average power is stable for a given model–hardware pair (<5% coefficient of variation) and insensitive to input resolution, image content, or prompt type.
- Measured relation (RTX 3070): P̄ ≈ 12.1 W × (model size in billions) + 42.2 W (R² = 0.918).
Time (not power) drives energy
- Energy E = P̄ × t, and P̄ is constant → all energy variation comes from variation in wall-clock time t.
Two-phase inference asymmetry
- Prefill (parallel) processes all input tokens in one pass (compute-bound, high arithmetic intensity).
- Decode (autoregressive) issues one forward pass per output token (memory-bandwidth-bound, low arithmetic intensity).
- Marginal cost per output token (αd) ≫ marginal cost per input token (αp).
Quantitative dominance of decode
- Decode accounts for 86–97% of energy across models profiled.
- Per-output-token cost is 11–39× per-input-token cost (range depends on model/size).
Visual-token pruning has limited energy upside
- For fixed-token architectures, removing all visual tokens can save at most ≈10% of total energy.
- By contrast, controlling output length can reduce total energy by up to ≈97% (depending on model scale and baseline output length).
Low-dimensional energy predictor
- A simple linear model using five features (model size, input token count, output token count, plus two interaction terms) explains ~98.6% of energy variance across profiled configurations — enabling accurate cross-model energy estimation without per-model calibration.
Image complexity effects
- Image complexity (object count) can change energy up to 4.1× at fixed resolution — but that variation is mediated by longer outputs for complex images, not by more expensive visual encoding.

Data & Methods

Models profiled (5): InternVL3-1B (Q8), InternVL3-2B (Q4_K_M), Qwen2-VL-2B (Q4_K_M), Qwen2.5-VL-3B (Q4_K_M), Gemma-3-4B (Q4_K_M). Architectures cover fixed-token and dynamic-token visual strategies.
Hardware platforms: NVIDIA RTX 3070 laptop GPU (8 GB, freq locked) and NVIDIA Jetson Orin NX (15 W mode).
Inputs and variation:
- Resolutions: 224², 448², 672², 896².
- Image complexity: COCO 2017 validation subset stratified into six object-count tiers.
- Prompts: short (one-word answer) vs open-ended descriptive prompts to elicit drastically different output lengths.
Inference stack and settings:
- llama-server (llama.cpp) serving; greedy decoding (temperature=0).
- Quantized weights as noted per model.
Measurement:
- Power samples every 100 ms (HWiNFO64 Pro for laptop total system power; INA3221 VDD_IN rail on Jetson).
- Per-inference average power = total energy / wall-clock duration; energy decomposition by measuring prefill and decode time/energy.
Modeling:
- Latency linearization: t ≈ αp·N_in + αd·N_out + β; αd ≫ αp measured per model.
- Cross-model linear energy predictor with five features achieved 98.6% explained variance.

Implications for AI Economics

Prioritize decoding-efficiency investments
- For edge/embodied deployments, optimizing vision encoders or visual-token reduction yields limited energy returns; reducing generated output (shorter replies, constrained generation) offers far larger energy and battery-life gains.
Product design & UX tradeoffs
- Features that increase average output length (verbose assistants, long captions, verbose explanations) have direct and large energy costs on-device — factor per-query energy into UX decisions, pricing, and battery budgeting.
Cost modeling & pricing
- Energy cost per query is roughly linear in output tokens (plus small input-token term) and scales with model size via the power fingerprint. This enables more accurate per-query cost forecasts and time-of-operation battery-cost accounting for deployed agents.
Model selection and scaling decisions
- Larger models increase base power and amplify decode dominance; scaling up model size increases marginal energy per output token and strengthens the case for output-control measures or hybrid architectures (e.g., small on-device model + remote large model).
Deployment & capacity planning
- The supplied low-dimensional energy predictor can be used to forecast operational energy consumption across workloads, informing battery sizing, charging cycles, maintenance schedules, and fleet operational economics for embodied devices (robots, drones, AR glasses).
Environmental and operational efficiency
- Emission reductions and operational cost savings are best achieved by mechanisms that reduce output length or reduce the number of decode steps (token compression, concise instruction-tuning, caching of standard replies, selective offloading) rather than focusing exclusively on visual-token pruning.
Hardware & systems implications
- Because decoding is memory-bandwidth-bound, edge-hardware optimizations that increase memory bandwidth or reduce KV cache footprint (e.g., more efficient KV representations, on-chip cache improvements) would yield outsized energy/latency benefits relative to compute-only improvements.
Research implications
- Efficiency research focusing on visual-token pruning or vision-only compression should be recalibrated to account for the limited energy impact; research that meaningfully reduces autoregressive decode cost (algorithmic, architectural, or systems-level) is likely to yield much bigger returns for edge VLM deployments.

If you want, I can: - Extract the exact numeric αp/αd values per model from the paper figures/tables, - Produce simple per-query energy estimators for your target model/hardware, - Suggest concrete algorithmic or system interventions (e.g., KV compression, early stopping heuristics, instruction tuning) with estimated energy savings.

Assessment

Paper Typedescriptive Evidence Strengthhigh — The paper presents direct, instrumented measurements of power and time across multiple models, architectures, input conditions, and two hardware platforms, and it isolates prefill vs decode phases to show that energy variation arises from time spent decoding (output tokens) rather than changes in average power draw. That combination of multi-factor experimentation and low-level profiling gives strong internal validity for the measured claims within the tested configurations. Methods Rigorhigh — The study systematically varies key factors (model, resolution, image complexity, prompt) and measures both instantaneous power and wall-clock time, enabling decomposition of energy = power × time and stage-level attribution (prefill vs decode). Coverage of multiple architectures and parameter scales (1B–8B) strengthens the analysis; the approach directly targets the mechanism (output-token-driven time) rather than relying on proxies. SampleFive vision-language models spanning three architecture families and parameter sizes from ~1 billion to ~8 billion; four input image resolutions; image set varied by object count (used to measure 'image complexity'); multiple prompt types; experiments run on two hardware platforms — NVIDIA RTX 3070 (desktop GPU) and NVIDIA Jetson Orin NX (edge) — with per-inference power and timing instrumentation to separate prefill and decode phases. Themesadoption productivity IdentificationSystematic empirical profiling: measure wall-power and wall-clock time for on-device VLM inference across controlled manipulations (model family/size, input image resolution, image complexity measured by object count, prompt type) on two hardware platforms; decompose inference into prefill (input token processing) and decode (output token generation) phases to attribute energy/time to input vs output tokens and compare effects across conditions. GeneralizabilityOnly two hardware platforms tested (one desktop-class GPU and one edge SoC); results may differ on other GPUs, TPUs, or future edge accelerators., Five models across three architectures provide breadth but do not cover all VLM designs (e.g., encoder-only VLMs, retrieval-augmented, or heavily quantized/compiled variants)., Measured for specific inference stacks/runtimes; optimizations, compiler toolchains, batching, mixed precision, quantization, or hardware-specific kernels could change power/time tradeoffs., Focuses on inference (not training) and on autoregressive decode behavior; models or use-cases with non-autoregressive outputs or constrained decoding may show different patterns., Image complexity operationalized as object count; other measures of visual complexity (scene semantics, resolution-dependent preprocessing) may have different effects.

Claims (8)

Claim	Direction	Outcome	Confidence & Evidence	Details
Average inference power is a model-intrinsic constant, invariant to input resolution, image complexity, and prompt type, with less than 5% variation across all conditions. Other	null_result	average inference power	Reading fidelity high Study strength high	n=5 less than 5% variation across all conditions 0.3
All energy variation across inputs must arise from variation in inference time, not from variation in power draw. Task Completion Time	positive	inference time (wall-clock time)	Reading fidelity high Study strength medium	n=5 0.18
Each output token costs 11 to 39x more wall-clock time than each input token due to the compute-bound and memory-bound asymmetry between prefill and decode, making output token count the dominant driver of both latency and energy. Task Completion Time	positive	wall-clock time per token (latency)	Reading fidelity high Study strength high	n=5 11 to 39x more wall-clock time 0.3
Image complexity, measured by the number of objects in an image, induces up to 4.1x energy differences at identical resolution. Other	positive	inference energy consumption	Reading fidelity high Study strength medium	up to 4.1x energy differences 0.18
The variation in energy with image complexity arises not from increased visual processing cost, but from differences in output length. Other	positive	cause of energy variation (output length vs visual processing cost)	Reading fidelity high Study strength medium	not reported 0.18
Even removing all visual tokens saves at most 10% of total energy for fixed-token models, exposing a fundamental limitation of visual token pruning. Other	negative	total inference energy	Reading fidelity high Study strength medium	n=5 at most 10% of total energy 0.18
Across models spanning 1 billion to 8 billion parameters, controlling output length saves up to 97% of total energy, with the energy dominance of decoding growing stronger at larger model scale. Other	negative	total inference energy savings from controlling output length	Reading fidelity high Study strength high	n=5 saves up to 97% of total energy 0.3
The true energy bottleneck in edge VLM inference is not what the model sees (visual input), but how much it says (output length). Other	positive	primary determinant of inference energy usage	Reading fidelity high Study strength high	n=5 0.3

Edge vision-language models consume power at a near-constant rate, so energy costs scale with how much the model outputs rather than what it sees; trimming visual tokens can save only a few percent, whereas controlling output length can cut energy use by up to 97%.