Rendering source code as compact images can sharply reduce provider-reported input tokens for some models — roughly cutting counts by three-quarters to nearly 90% — but billing behavior is provider-dependent, with Gemini charging more for small files and only crossing below text counts at larger sizes; model aliases collapse to about five distinct accounting signatures rather than representing independent, uniform token accounting.

Main Finding

For a fixed compact rendering pipeline and five pinned source files, sending code as PNG images produced substantially lower provider-reported input-token counts in aggregate — but with large provider- and size-dependent differences. Across paired measurements (675 text/image pairs, 1,350 API calls), weighted image/text ratios were: - Anthropic: 0.135 (86.5% reported input-token reduction) - OpenAI: 0.194 (80.6% reduction) - Gemini: 0.242 (75.8% reduction)

However, this aggregate masks heterogeneous behavior: Anthropic and OpenAI images were lower at every tested size, while Gemini images incurred far higher reported tokens for short snippets (≈6.95× at 20 lines) and only crossed below text at ~200 lines. The study measures provider-reported token accounting only — not fidelity, compute, latency, or dollar cost.

Key Points

• Scope & scale
  • Reproducible measurement: 1,350 successful API calls across Anthropic (4 aliases), OpenAI (6 aliases), and Google Vertex AI / Gemini (5 aliases).
  • 5 pinned source files (Python, JavaScript, Rust, Go, Java) sliced into 9 nested prefix sizes: 20, 50, 100, 200, 400, 800, 1,200, 1,600, 2,000 lines.
  • 675 complete text/image paired observations (same file/size/instruction).
• Compact-image pipeline
  • Two-stage transform: (1) compact indentation markers (replace 4-space indent with a marker and encode runs of spaces compactly), (2) render to PNG pages (pxpipe-proxy).
  • Both arms append identical one-sentence instruction: “Summarize what this code does in one sentence.”
• Main quantitative findings
  • Weighted-by-token aggregate ratios (image tokens / text tokens): Anthropic 0.135, OpenAI 0.194, Gemini 0.242.
  • Size dependence: Anthropic and OpenAI remain below parity at all tested sizes; Gemini > parity for small sizes (ratios ≫1 at 20–100 lines), first crosses below parity at 200 lines, then declines with size.
  • Aggregation sensitivity: token-volume-weighted metric (used as headline) vs equal-cell unweighted mean diverge — e.g., Gemini’s unweighted mean >1 (images worse) while weighted mean <1.
• Heterogeneity & anomalies
  • Alias-level clustering: some aliases share identical accounting signatures (suggests shared internal rules), while others differ (e.g., Claude Haiku 4.5 higher; Gemini 2.5 Flash higher).
  • Workload descriptives: modest variation across the five files (ranges reported), but files are not independent draws from a language population.
  • Non-monotonic accounting observed: a targeted audit showed Gemini image token counts change non-monotonically across a page boundary (800-line one-page request > 2,000 image tokens; 1,200-line two-page request only ~516 image tokens), demonstrating page-sensitive discontinuities.
• Limitations emphasized by the authors
  • The measurement is black-box and narrow: does not assess semantic fidelity, OCR/transcription accuracy, task success, compute consumed, latency, or monetary cost.
  • The treatment conflates visual modality and compact indentation markers (no compact-text-only control).
  • Corpus design: single pinned file per language; nested prefixes overlap and are not independent samples.

Data & Methods

• Corpus and sampling
  • One revision-pinned source file per language (CPython asyncio/base_events.py; Lodash JS; Rust compiler parser; Go net/http/server.go; Guava LocalCache.java).
  • Extracted nested prefixes at 9 sizes: 20–2,000 lines.
• Treatment and controls
  • Raw-text arm: snippet as-is + instruction.
  • Compact-image arm: transform indentation → render to PNG pages → same instruction.
  • No image-only vs compact-text-only isolations; transformation changes both modality and representation.
• Providers, models, and calls
  • Anthropic: Messages API; 4 aliases; 360 successful calls; 180 complete pairs.
  • OpenAI: Responses API (image detail high); 6 aliases; 540 successful calls; 270 complete pairs.
  • Gemini (Vertex AI): generate_content; 5 aliases; 450 successful calls; 225 complete pairs.
  • Total recorded: 1,350 successful calls; 675 paired (text+image) observations.
• Metrics and aggregation
  • Primary per-pair ratio r_i = image_input_tokens / text_input_tokens.
  • Aggregated weighted ratio R_G = (sum image tokens in group) / (sum text tokens in group) — weights by token volume (authors argue this answers the transmitted-context-volume question).
  • Sensitivity: also report unweighted mean of per-cell ratios; results differ materially (fixed costs dominate).
• Validation & reproducibility
  • Validator rejects duplicates, ensures consistency across records; all raw responses, scripts, corpus spec, validators, and analysis are released in artifact repository: https://github.com/ron-42/code-image-token-accounting.

Implications for AI Economics

• Token counts ≠ dollars or compute
  • Provider-reported “input tokens” are an accounting signal used for quotas/pricing, but they do not necessarily map 1:1 to provider compute, inference cost, or billing. Providers may price image vs text tokens differently, apply fixed per-request fees, or have internal offsets/caching.
  • Therefore a lower reported token count is necessary but not sufficient evidence of cost savings.
• Fixed costs and break-even sizing matter for routing and cost optimization
  • The strong fixed-cost behavior (large per-request or per-image minima) implies visual conversion is beneficial only above provider- and pipeline-specific break-even sizes. For example, with this pipeline Gemini only becomes advantageous at ≥200 lines, while Anthropic/OpenAI were advantageous at all tested sizes.
  • For economic routing (deciding representation to minimize cost/latency), one must compute per-provider, per-model break-even thresholds, not assume universal gains.
• Opportunities and risks for operational arbitrage
  • Sending images to reduce reported input tokens could (temporarily) lower consumption of token-based quotas or billed units. This creates potential arbitrage/optimization opportunities for organizations constrained by token quotas or per-token pricing.
  • Risk: providers can change accounting rules, rate structures, or detection policies; relying on such a tactic is brittle and may invite policy changes or penalties.
• Need to connect accounting to end-to-end value
  • Beyond reported token reduction, firms must evaluate:
    • Task success rates with image representation (e.g., code edit accuracy, test passes).
    • Latency and bandwidth trade-offs (PNG rendering, upload size).
    • Actual pricing differences (image vs text token price, per-request fees).
    • Effects on caching and downstream agent efficiency.
  • Only a joint frontier of usable information, latency, and cost determines whether visual conversion is economically rational.
• Practical recommendations for practitioners and researchers
  • Calibrate and re-run per-provider accounting measurements periodically (aliases and rules change).
  • Use mixed strategies: keep short/active spans in text; consider images for large, read-mostly context if fidelity is acceptable.
  • Compare visual conversion against alternative, smaller-changes: retrieval, textual compression, prompt caching — those preserve text-native interfaces and may be cheaper or more robust.
  • When reporting savings, publish size-stratified break-even analyses and the chosen aggregation estimand (token-weighted vs configuration-weighted).
• Policy and vendor considerations
  • Providers should document how multimodal inputs are tokenized/accounted (per-image minima, page partitioning, resolution thresholds) — transparency would reduce uncertainty and enable rational cost optimization.
  • For economic modeling and forecasting, firms must treat provider-reported token units as a changing policy variable, not a stable hardware cost proxy.

Suggested next steps to close the economics loop - Replicate with: multiple independent files per language; compact-text-only controls; alternative renderings; and explicit measurements of monetary price, elapsed latency, bytes transferred, and downstream task quality (e.g., exact code recovery, tests passed). - Model the full cost function (billing + latency + task failure penalty) to derive operational routing rules (per-provider, per-model, per-file-size).