Fine-tuned open-source vision–language models can outperform zero‑shot commercial multimodal LLMs on templated check extraction—reaching F1≈0.985 with 3k examples—while latency, cost and governance constraints change which approach is optimal, so the authors release an evaluation harness and a simple TCO-based practitioner framework to guide selection.

Main Finding

Fine-tuning modestly sized open-source vision-language models (VLMs) on ~3K labeled templated-document examples yields large operational gains: the best fine-tuned open-source models (Qwen2.5-VL 3B/7B) surpass every zero-shot commercial multimodal LLM on a synthetic check-extraction task (F1 ≈ 0.983–0.985 vs. best commercial GPT-5 F1 = 0.928). Practical approach selection must trade off accuracy, latency, governance, volume and total cost; the authors provide a lightweight, measurement-grounded selection framework that maps task constraints to a cost-optimal feasible model.

Key Points

• Dataset & task: 3,748 synthetic check images (2,998 train; 750 held-out test) with four output fields (Payee, Amount, Date MM/DD/YYYY, Bank (Chase/Citi/BOA)). Prompt/schema identical across models; models must output JSON.
• Models evaluated: 11 systems
  • Commercial/reasoning: GPT-4.1 (vision), OCR+GPT-4.1, Azure Content Understanding (ACU), GPT-5, Claude Sonnet 4.5
  • Open-source VLMs (pretrained and fine-tuned): Qwen2.5-VL (3B, 7B) LoRA; SmolVLM2 (2.2B, 500M, 256M) full FT
  • Non-LLM baseline: OCR → regex
• Headline accuracy (held-out 750-doc pool)
  • Best fine-tuned open-source: Qwen2.5-VL-7B FT F1 = 0.985; Qwen2.5-VL-3B FT F1 = 0.983
  • Best commercial/reasoning: GPT-5 F1 = 0.928; OCR+GPT-4.1 F1 = 0.917; GPT-4.1 (vision) F1 = 0.909; ACU = 0.889; Claude Sonnet 4.5 = 0.857
  • Non-LLM OCR→regex floor: F1 = 0.395
• Latency (single-document p95 approximations; measured on A100 or vendor APIs)
  • Qwen-7B FT: ~3.5 s; Qwen-3B FT: ~4.2 s
  • GPT-4.1 vision: 12.2 s; OCR+GPT-4.1: 18.1 s; GPT-5: 26.3 s
• JSON parse reliability: small SmolVLMs had high JSON failure pre-FT; fine-tuning reduced failures but sub-500M models remain fragile.
• Per-field: Date is the hardest field for most commercial models (Date F1 as low as 0.468 for Claude Sonnet 4.5); Bank (closed 3-way vocab) is trivially high for larger models; fine-tuned open-source models learn to localize/copy Date, reducing over-extraction.
• Fine-tuning impact: large absolute gains for smaller models (e.g., Smol2.2B from F1 0.504 → 0.906), and converts Smol-500M/-256M from near-zero to usable outputs.
• GPU sensitivity: moving from A100 → H100 gives ~1.4–1.7× speedup with negligible accuracy change; model throughput/latency depends on GPU choice and architecture (3B vs 7B tradeoffs).
• OOD & cross-task:
  • Some fine-tuned models retain much of in-domain performance on external check data (Smol2.2B FT retained ~87% of in-domain F1).
  • Training-data artifacts can cause catastrophic OOD failures (e.g., Qwen-3B FT predicting a fixed year).
  • On a 36-field insurance form task (4k train / 1k test), fine-tuned open-source models beat zero-shot GPT-4.1 on checkbox fields (e.g., Smol-500M FT checkbox accuracy 0.99 vs GPT-4.1 0.48).
• Error taxonomy: (E1) degenerate repetition/looping on small models (JSON failures), (E2) phonetic payee misspellings on mid-size models, (E3) date over-extraction in commercial LLMs — each suggests targeted post-processing and reliability engineering.

Data & Methods

• Synthetic dataset: template-driven renderer with controlled perturbations; train/test split at source-table row level (no overlapping field values), separate background pools for train/test; 2,998 train samples, 750 held-out test.
• Fine-tuning recipes:
  • Qwen 3B/7B: LoRA adapter (r=16, α=32), small % trainable params, multiple epochs (3 for 3B; 5 for 7B with gentler LR)
  • SmolVLMs: full fine-tuning (3 epochs for 2.2B, etc.), paged AdamW-8bit
• Metrics:
  • Micro-entity F1 (aggregate over fields), per-field F1, document-level accuracy (all 4 fields correct), JSON parse success, latency (p95 approximated from measurements), error-mode classification.
• Candidate set and selection framework:
  • Characterise each candidate m by Q(m) (quality metrics), p95 latency Lp95(m), governance capabilities G(m), development cost Cdev(m), per-document inference cost cinf(m), fixed monthly cost cfixed(m).
  • Feasible set MF: models satisfying Qi >= Qi,min, Lp95 <= Lmax, G(m) ⊇ Greq.
  • Cost-optimal choice: minimize total cost = Cdev(m) + T * (V * cinf(m) + cfixed(m)) over MF.
  • Breakeven volume between two models m1, m2: V* = (ΔCdev + T Δcfixed) / (T (cinf(m2) − cinf(m1)))
• Illustrative case study (demonstration, not production guidance):
  • Task profile: V = 100K docs/month, T = 12 months, Qmin = 0.85 F1, Lmax = 5 s, governance = self-hosted only
  • Cost assumptions (authors’ illustrative estimates):
  • API per-doc: GPT-4.1 ≈ $0.004/doc; GPT-5 ≈ $0.025/doc; Claude ≈ $0.006/doc; ACU ≈ $0.01/page; OCR-only ≈ $0.0015/doc
  • Self-hosted fixed: AWS g5.xlarge (A10G) cost ≈ $734/month (dedicated provision); marginal per-doc negligible until saturation
  • Development: labeling 2,998 samples ≈ $6K ($2/doc), engineering ≈ $7.5K, production overhead $1.5–3.5K
  • Hardware projection for p95 latencies: A10G slower than A100 by ~1.7× (approximate)
  • Using these inputs the framework filters by Qmin/Lmax/Greq then minimizes total cost to pick the recommended candidate for that scenario.

Implications for AI Economics

• Fine-tuning is high-return for templated extraction tasks:
  • A modest annotation budget (~3K examples) and modest engineering (~$10–20K) can push open-source VLMs past zero-shot commercial accuracy while reducing latency and avoiding per-call API fees. This makes fine-tuning attractive for mid-to-high volumes or strict latency/governance constraints.
• Cost trade-offs (APIs vs self-hosted open-source):
  • APIs: low up-front dev cost, pay-as-you-go per-document pricing; attractive for low volume, short pilots, or when governance allows. Per-doc costs for powerful reasoning models (e.g., GPT-5) can be large and dominate TCO at scale.
  • Self-hosted fine-tuned models: higher up-front labeling & engineering cost and fixed infra cost, but much lower marginal per-doc cost. Breakeven volume depends on ΔCdev and per-doc price gaps; the provided V* formula lets practitioners compute it.
• Governance constraints materially change the feasible set:
  • If data-locality or contractual constraints mandate self-hosting, some top API options are excluded; fine-tuned open-source models become more attractive even at moderate volumes.
• Latency and hardware choices affect economics:
  • Smaller open-source models (3B–7B) can deliver low p95 latencies on modern GPUs and, after fine-tuning, can both meet strict latency ceilings and reduce TCO versus API calls that have higher wall-clock latency.
  • GPU type (A100 vs H100 vs A10G) changes throughput and thus the size/number of instances required; hardware selection should be part of cost modeling.
• Operational costs beyond raw model metrics:
  • Error modes (JSON failures, over-extraction, spelling errors) require post-processing, monitoring, human-in-the-loop validation, and retraining pipelines — these are non-negligible operational expenditures that the simple TCO model must be extended to capture in production.
• Domain shift risk:
  • Synthetic/template training and strong in-domain performance do not guarantee OOD robustness; model retraining, schema validation, and data-quality controls add recurring costs and risk. Training-data artifacts can cause catastrophic OOD behaviors and must be guarded against.
• Strategic recommendation (practical rules of thumb):
  • If governance allows self-hosting and expected monthly volume is moderate-to-high (or Lmax small), invest in ~3K labels + fine-tune a 3B–7B open-source VLM (likely lower TCO and higher accuracy/latency).
  • If governance forbids self-hosting or volume is very low, use API models to avoid development overhead.
  • For very constrained hardware budgets or on-device needs, small models can be viable after careful robustness engineering, but expect fragility and added validation cost.
• Caution: synthetic/template dataset limits generalisation. The study is measurement-grounded and gives a practical framework, but practitioners should re-run the supplied harness on their own data to populate model metrics before making procurement decisions.

Overall, the paper demonstrates that (i) fine-tuning relatively small VLMs is a cost-effective lever for improving extraction performance and meeting latency/governance constraints, and (ii) selection should be driven by a constrained TCO optimization that includes dev cost, per-doc pricing, latency budgets, governance filters, and domain-shift risk.