A large-scale reliability audit shows mid-sized open-weight LLMs can deliver near-production performance for SME agent deployments while smaller models fail mainly at tool initialization. qwen2.5:32b matched GPT-4.1’s flawless results and qwen2.5:14b offered a 96.6% success rate with 7.3s latency, suggesting affordable, reliable on-prem agent deployments are feasible.

Main Finding

A diagnostic framework (12-category error taxonomy + evaluation protocol) reveals that procedural reliability in multi-agent LLM systems is primarily limited by tool-initialization failures. Across 1,980 deterministic invoice-reconciliation test cases, open-weight models reach closed-source parity only at ~32B parameters (qwen2.5:32b = 100% SR matching GPT-4.1). A 14B open-weight model (qwen2.5:14b) delivers a practical SME-ready trade-off (≈96.6% success, 7.3 s latency on RTX A6000; recommended on RTX 4090 for cost-sensitive deployments).

Key Points

• Diagnostic contribution
  • 12-category taxonomy: 4 error types (Not Initialized, Arguments Mismatch, Error, Result Mismatch) × 3 tool categories (OCR, DB Query, DB Update).
  • Focus moves beyond aggregate success rates to actionable failure causes.
• Empirical reliability thresholds
  • qwen2.5:32b: 100% success (parity with GPT-4.1).
  • qwen2.5:14b: 96.6–97.4% success — identified as minimum viable production configuration for many SMEs.
  • Smaller models (7B, 3B) suffer catastrophic initialization failures (qwen2.5:3b ≈ 13–15% SR).
• Dominant failure mode
  • Tool initialization failures (e.g., DB_UPDATE_TOOL_NOT_INITIALIZED, DB_QUERY_TOOL_NOT_INITIALIZED) dominate error distributions across models and tasks.
  • Qualitative breakdown: omission failures (~68% of sampled initializations) and malformed-call failures (~32%).
• Hardware and latency
  • Up to 8.2× latency variation across tested hardware (RTX A6000, RTX 4090, Apple M3 Max).
  • Example: qwen2.5:14b runs in 7.3 s on RTX A6000 but ≈60 s on M3 Max.
• Practical deployment tiers (authors’ recommendations)
  • Maximum reliability: qwen2.5:32b on RTX A6000.
  • Balanced (SME) : qwen2.5:14b on RTX 4090 (≈$5K).
  • Budget-constrained: <14B models require extra validation, retries, or orchestration to be production-viable.
• Reproducibility
  • Deterministic evaluation (temperature=0), 4-bit quantization (Q4_K_M) for open-weight models; code/datasets available at the cited GitHub repo.

Data & Methods

• Use case and architecture
  • Task: invoice reconciliation (multi-modal; vision and text).
  • Agent architecture: three specialized agents (Email/OCR, Data Engineering (DB Query & Update), Reconciliation) orchestrated with LangGraph; instrumentation captures tool call traces.
• Dataset
  • Synthetic, domain-mirroring dataset: 1,980 deterministic test instances, balanced vision (990) and text-only (990).
• Models & deployment configurations
  • Closed-source baselines: OpenAI GPT-4 variants (gpt-4o, gpt-4.1 family) and Anthropic Claude (3.5/3.7).
  • Open-weight models: qwen2.5 series (3B, 7B, 14B, 32B, 72B) and Functionary (8B, 70B).
  • Open models run locally with Ollama v0.6.8 and 4-bit quantization; hardware: NVIDIA RTX A6000 (48GB), RTX 4090 (16GB), Apple M3 Max (96GB).
• Metrics and protocol
  • Metrics: Success Rate (SR), Execution Time, Process Steps, OCR F1 Score.
  • Protocol: deterministic runs (temperature=0), fixed prompts, all runs instrumented for fine-grained error labeling per taxonomy.
• Qualitative analysis
  • Manual inspection of 200 failure cases from small/medium models to classify omission vs malformed-call patterns and link to procedural reasoning gaps.

Implications for AI Economics

• Cost vs reliability trade-offs
  • Achieving closed-source parity is feasible with large open-weight models, but hardware procurement and operating costs (e.g., RTX A6000 ≈ $10K) matter. SMEs can reach near-production reliability with 14B models on mid-range GPUs (RTX 4090 ≈ $5K).
  • Cloud API usage (closed-source) may offer lower up-front hardware cost but raises recurring API expenditure and data-sovereignty concerns; open-weight models shift costs from per-call fees to capital + maintenance and ops.
• Effective low-cost interventions
  • Many failures are procedural (tool-init), not purely model-capacity limits. Engineering investments—better orchestration, schema-grounding, explicit function-call scaffolding, input validation, and lightweight verification/retry policies—may yield substantial reliability gains at lower marginal cost than scaling models.
  • For budget-constrained orgs, adding orchestration/validation layers is likely more economical than upgrading model capacity to 32B.
• Throughput, latency, and operational efficiency
  • Latency variation (up to 8.2×) affects throughput and staffing/productivity costs. Faster hardware reduces per-task wall time, enabling higher throughput and lower per-task operational cost.
• Risk & compliance economics
  • Open-weight local deployment enables data sovereignty and potentially lower compliance/legal risk for privacy-sensitive workflows, which has monetary value (reduced regulatory overhead, liability).
• Deployment decision framework for practitioners
  • If SLA-critical and privacy-sensitive: invest in ~32B open-weight + capable GPU or use closed-source premium API.
  • If cost-sensitive with moderate SLAs: deploy ~14B on mid-range GPUs plus robust orchestration/validation.
  • If severely cost-limited: expect higher operational monitoring/verification costs; invest in orchestration to mitigate high failure rates of <14B models.
• Research/economic priorities
  • Prioritize engineering research that reduces initialization failures (tool-call schemas, verifier modules, low-cost orchestration) because these may produce high ROI by lowering the required model scale and associated hardware cost.

If you want, I can produce a one-page decision checklist for SMEs (cost estimate, recommended stack, minimal orchestration patterns) based on the paper’s results.