A large-scale benchmark finds that client-side image preprocessing for cloud VQA is a double-edged sword: the same crop, compression, or grayscale trick can either cut bills and speed responses or raise token costs, increase latency and reduce answer accuracy, depending on the provider’s tokenization and model stack.

Main Finding

Client-side image preprocessing for cloud-driven vision-language VQA is not a free win. Its effect on accuracy, latency, bandwidth, and monetary token cost depends strongly on (a) which commercial VLM and provider are used, (b) the provider’s image-token accounting rule, (c) the API/streaming paradigm and task formulation, and (d) what visual content is removed. A wrong preprocessing choice can increase cost and latency while degrading accuracy; the right choice can meaningfully reduce cost with little accuracy loss — but this is model- and scenario-specific.

Key Points

• Scope and scale: VQABENCH is the first benchmark treating client-side input preprocessing as a controlled variable for proprietary cloud VLM-based VQA. It evaluates 12 preprocessing techniques across three complementary public VQA datasets, tested on four commercial VLMs from three providers, totalling 95,168 API calls.
• Measured dimensions: For every request the benchmark jointly measures four end-to-end quantities — answer quality (accuracy), payload size (bytes transmitted), token usage (text/image/output/cached as reported by providers), and latency (broken down into local preprocessing, upload time, server/API processing, and response streaming).
• Preprocessing techniques: The assessed methods include global reductions (compression like JPEG/WebP, downsampling, grayscale), and region-selective methods (saliency crops, object-detection crops, gaze-guided crops, thumbnails + crops, etc.). These are practical client-side optimizations available to developers who cannot change server internals.
• Major empirical findings:
  • Preprocessing is not universally beneficial — effects vary by model, provider, API paradigm, and task.
  • Latency improvements are dominated by the model–provider stack and task formulation; preprocessing is a secondary factor.
  • Bytes transmitted (payload) do not map directly to image tokens: image-token counts depend on decoded pixel geometry (the pixel grid), not file size. A heavily compressed file that decodes to the same pixel dimensions will incur the same image-token cost as a larger file.
  • Spatial reduction (cropping/downsampling) reduces image tokens only under pixel-proportional token-accounting schemes. Under flat-rate or tile-based tokenization used by some providers, spatial cropping can fail to reduce tokens and may even increase token counts substantially.
  • Region-selective methods can increase image-token usage (reported increase of over 400% on average in some settings) if they change decoded geometry or force additional tiles/units in provider tokenization rules.
  • What gets removed matters more than how much: removing color often has limited accuracy impact, while removing task-relevant regions (even with similar total pixel reductions) can cause large accuracy drops.
  • Accuracy effects can reverse across models: the same preprocessing that helps one VLM can harm another.
• Transparency and replication: The authors release benchmark code and measurement artifacts (link in paper) to support reproducibility.

Data & Methods

• System model: Benchmarked a realistic cloud-VLM-based VQA pipeline where a client captures an image and question, optionally preprocesses the image, encodes it (JPEG/WebP + Base64 in a JSON request), uploads to a proprietary VLM API (REST/streaming/Realtime), and receives a textual answer. The benchmark instruments both client-observable timings and provider-returned token usage.
• Metrics and decomposition:
  • Payload size = encoded image bytes + Base64 inflation (~33%) + JSON/request overhead.
  • Token usage = text input tokens, image input tokens, output tokens, cached tokens, and total tokens (what providers return in usage records). Important: image tokens derive from pixel dimensions after decoding, not from transmitted bytes.
  • Latency decomposed into local preprocessing time, upload time, server-side time-to-first-response, and first-response-to-completion.
  • Answer quality measured as task-appropriate VQA accuracy on the chosen datasets.
• Experimental design:
  • 12 client-side preprocessing techniques spanning global and region-selective reductions.
  • Three public VQA datasets chosen to be complementary (egocentric/wearable and general VQA scenarios).
  • Four commercial VLMs from three providers (selected from a broader set of models) to capture different token-accounting rules and API behaviors.
  • Total of 95,168 API calls to obtain robust measurements across the full performance space.
• Provider token-accounting analysis: The benchmark isolates provider-specific tokenization/accounting rules (pixel-proportional vs flat-rate/tile-based) and shows how they mediate whether preprocessing changes billed tokens.

Implications for AI Economics

• Monetary costs depend on tokens, not just bytes. Developers optimizing for cost must measure billed token usage (image + text + output) rather than network traffic alone. A smaller JPEG file doesn’t necessarily reduce billed image tokens.
• Provider pricing & tokenization policy materially affect deployed costs and optimal client-side strategy. Flat-rate or tile-based image token accounting can negate or invert expected savings from spatial cropping or region selection; thus procurement and API choice are economically consequential.
• Latency and user experience are determined more by the provider/model stack and API paradigm than by client preprocessing. For applications with strict latency budgets (e.g., assistive tech, AR), the economic trade-offs should consider provider SLA/performance and streaming behavior, not preprocessing alone.
• Region-selective preprocessing can reduce compute/token costs if the provider uses pixel-proportional tokenization, but can increase costs under other accounting rules — potentially increasing per-request billing by hundreds of percent. Deployers must test preprocessing strategies specifically on the target provider/model.
• Pricing transparency and tooling matter: providers exposing clearer modality/token billing and image-token rules would enable more predictable cost-accuracy trade-offs and better economic efficiency across VQA applications.
• Business and product guidance:
  • Empirically validate preprocessing on each target VLM and dataset before deploying; do not assume universal benefits.
  • Prefer strategies that preserve task-relevant content rather than maximally shrinking pixels (e.g., keep color if it matters little but avoid removing focal objects).
  • Consider negotiating or selecting providers whose token-accounting aligns with your optimization objective (e.g., pixel-proportional tokenizers if you plan to use region-selective cropping).
  • For cost-sensitive high-volume services, consider hybrid architectures (some on-device lightweight processing + selective cloud calls), or push for server-side token reduction capabilities from providers.
• Research & policy directions with economic relevance:
  • Design adaptive preprocessing that predicts both accuracy loss and billed-cost change for a given provider/model to enable per-request economic optimization.
  • Advocate for standard, transparent billing primitives for multimodal tokens to reduce uncertainty for deployers and enable fairer pricing models for visual AI services.

If you want, I can: - Extract the benchmark’s per-technique quantitative effects (payload, token, latency, accuracy) as tables or plots (requires the full results section), or - Translate the findings into a short checklist for deploying a cost-aware mobile VQA product on a specific provider.