0 cumulative citations
View corpus contextExposing domain-aligned parameterized tools rather than raw SQL lets small local LLMs match or beat larger models’ accuracy while slashing latency and cost per correct answer; in a reproducible Sakila benchmark a verticalized pack achieved pooled accuracy 0.939 versus 0.666 for raw SQL and reduced tokens-per-correct-answer by 2–12×.
From SQL Generation to Tool Selection: A Domain-Oriented Pattern for MCP Servers
Exposing curated, server-side parameterized domain tools instead of a generic execute_sql interface yields much higher correctness, lower latency, and far lower cost per correct answer across small-to-medium local LLMs in a reproducible Sakila benchmark.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Agents built on Large Language Models (LLMs) increasingly reach enterprise data through the Model Context Protocol (MCP), and many MCP database servers maximize flexibility by exposing a single generic SQL execution tool. This paper proposes the Domain-Oriented Tooling Pattern: instead of generating SQL at query time, the model selects from a small set of domain-aligned tools whose parameterized queries encapsulate schema navigation, joins and business rules on the server side. We formalize the pattern around three architectural invariants and introduce Model Demotion, the observation that replacing SQL synthesis with intent classification lowers the model tier required to serve routine requests. As a reference implementation we present MCP Blueprint, an open-source framework in which domain tools are defined declaratively as YAML metadata plus external parameterized SQL files. We evaluate the pattern with a public reproducibility benchmark comparing three MCP server designs - raw SQL execution, a thin generic tool pack, and a verticalized domain pack - on four local models (3B-8B) across seventeen customer-facing tasks over the Sakila database (609 completed cells; temperature 0; three repetitions per cell). The verticalized pack reaches a pooled mean score of 0.939 versus 0.666 for raw SQL and 0.605 for the generic pack; the smallest model improves from 0.583 to 0.929, matching or exceeding every larger configuration while cutting cost per correct answer by an order of magnitude. All harness code, prompts, gold answers, frozen packs and per-cell results are publicly available.
Summary
Main Finding
Replacing on-the-fly SQL generation with a small set of domain-aligned, server-side parameterized tools (the "Domain-Oriented Tooling Pattern") markedly improves accuracy, latency, determinism and cost for routine enterprise retrieval tasks. Encapsulating schema navigation, joins and business rules on the server lets much smaller models (SLMs) serve the routine majority of requests—what the paper terms "Model Demotion"—and yields large reductions in tokens/time per correct answer versus raw SQL or poorly designed generic tool packs.
Key Points
- Domain-Oriented Tooling Pattern: expose domain operations (e.g., customer_account_summary, recommend_films) as typed tools backed by pre-authored parameterized SQL, rather than a single generic execute_sql tool. This moves schema knowledge, joins and business rules server-side and gives the model an intent-classification + slot-filling task.
- Model Demotion: converting open-ended SQL synthesis into intent classification and slot-filling lowers the required model tier for routine requests; small local models can match or exceed larger models that must synthesize SQL.
- Tool design matters: a well-verticalized pack (domain tools with rich parameter contracts and embedded business logic) substantially outperforms both raw SQL and a shallow/generic tool pack. Exposing tools poorly can be worse than raw SQL.
- Operational benefits: reduced context overhead, fewer probabilistic execution risks (bad queries), smaller security surface (bounded operations), more deterministic behavior, and easier failure-mode handling.
- Work partitioning: routine majority served deterministically by packs + small models; exceptional tail handled by human-in-the-loop to author new pack operations (continuous improvement loop).
Data & Methods
- Reference implementation: MCP Blueprint — declarative packs (YAML + external parameterized SQL) separated from an engine layer (transports, pooling, validation). Code & artifacts: https://github.com/meob/mcp-blueprint and benchmark repo https://github.com/meob/mcp-blueprint-benchmark (CC BY 4.0).
- Benchmark design:
- Three server configurations over the same Sakila sample DB:
- A: Raw SQL (single execute_sql tool + schema in prompt).
- B: Verticalized pack (5 domain tools encapsulating joins/business rules).
- C: Generic thin-tool pack (table-oriented, minimal descriptions).
- Models (local, tool-calling enabled): llama3.2:3b, qwen2.5:3b, qwen2.5:7b, llama3.1:8b.
- Tasks: 17 enterprise-style tasks (customer lookup, rental state, recommendations, catalog details, workflows, edge cases). 4 models × 3 approaches × 17 tasks × 3 repetitions ≈ 612 planned cells (609 completed).
- Protocol: temperature 0, seed 42, context 8192, ≤10 agent steps, 3 repetitions per cell.
- Scoring: deterministic rule-based checks vs live-computed gold answers (no LLM judges). Recorded per-cell token counts, latency, agent steps, tool-call traces, and per-check results.
- Three server configurations over the same Sakila sample DB:
- Main quantitative results (pooled across models):
- Mean scores: B (verticalized) = 0.939; A (raw SQL) = 0.666; C (generic) = 0.605.
- Fully-correct cells: B 85% (174/204), A 33% (67/201), C 31% (63/204).
- Latency (mean per cell): B 4.4 s, C 6.4 s, A 17.3 s.
- Tokens per cell (mean): B 3,056; C 2,894; A 3,953.
- Cost/efficiency (tokens per correct answer): B far lower than A or C. Examples:
- llama3.2:3b tokens/correct: A 31,476 → B 2,723 (≈11.6× reduction).
- qwen2.5:3b tokens/correct: A 17,467 → B 4,766 (≈3.7×).
- qwen2.5:7b: A 8,464 → B 4,331 (≈2.0×).
- llama3.1:8b: A 5,808 → B 2,531 (≈2.3×).
- Smallest model benefits most: llama3.2:3b improves from 0.583 (raw SQL) to 0.929 (verticalized).
- Per-task pattern: greatest gains on recommendation + negative-filtering and multi-tool workflows; simple single-query tasks converge across approaches.
Implications for AI Economics
- Lower inference cost: By enabling small local models to solve routine retrievals, domain-oriented tooling reduces tokens and latency per correct answer dramatically (2–12× in the study). This lowers operating costs (inference bill, latency-sensitive SLAs) and enables cheaper deployment tiers.
- Capital and provisioning implications: Organizations can avoid or postpone expensive frontier-model API use for a large fraction of traffic, shifting to SLMs hosted locally or on cheaper inference tiers for routine queries while reserving higher-cost models for exceptional/analytical tasks.
- Engineering trade-offs and ROI: The pattern requires upfront engineering (designing/authoring domain tools and parameterized SQL) and ongoing maintenance (extending packs as the domain evolves). However, the benchmark shows that this upfront cost can be repaid via much lower per-request inference expense and fewer production incidents from pathological SQL.
- Risk management and predictability: Encapsulating business rules server-side improves determinism, simplifies auditing and reduces the security/exposure surface compared to exposing raw SQL. This lowers operational risk and simplifies compliance/audit costs.
- Product design and labor allocation: The 95/5 partitioning (routine vs exceptional) formalizes a continuous-improvement loop where human domain engineers encode recurring cases once, shifting model invocations to cheaper, more reliable execution—changing the labor mix from repeated prompt engineering to pack authoring and governance.
- Cautions for economic generalization:
- Applicability is strongest for bounded, recurring retrieval workflows; exploratory analytics and open-ended queries still need expressive models or human workflows.
- Gains depend on tool quality: poorly designed generic tools can underperform raw SQL, so economic benefits require investment in good interface design and domain modeling.
- Benchmark limitations: experiments used the Sakila sample DB and local models; real enterprise schemas (larger, messier) could change absolute magnitudes though not the qualitative direction.
Practical takeaway: investing engineering effort to verticalize common domain operations and push schema- and rule-logic into server-side parameterized tools is likely to be highly cost-effective: it reduces the need for large-model inference on routine traffic, lowers latency and improves reliability—turning model capacity into an intent-classification service rather than a full SQL generator.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports a controlled, reproducible benchmark comparing three MCP server designs across four local LLMs and 17 tasks with rule-based gold answers and multiple repetitions; this provides strong internal evidence that domain-oriented tooling improves accuracy, latency and token cost in the tested setup. However, the evidence is limited to a single sample database (Sakila), a curated task suite, and locally served models, so external validity to production enterprise schemas, broader task distributions, or economic outcomes is limited.
Methods Rigormedium — The experimental design is systematic: multiple approaches (raw SQL, generic pack, verticalized pack), multiple models, fixed protocol (temperature, seed, context), rule-based scoring against live-computed gold answers, three repetitions per cell, and full reproducibility artifacts released. Weaknesses include reliance on a single toy/benchmark database (Sakila) rather than diverse enterprise schemas, a manually curated task set that may emphasize the proposed pattern, limited model diversity (local models only, some builds excluded), and absence of field or longitudinal deployment data.
SampleSakila sample PostgreSQL database (six task-relevant tables listed), 17 enterprise-style retrieval and workflow tasks (customer lookup, rental state, recommendations, catalog details, multi-step workflows, edge cases), three MCP server designs (A: raw execute_sql with DDL in prompt; B: verticalized domain pack with 5 domain-aligned tools; C: generic thin-tool pack), four instruction-tuned local LLMs served via Ollama (llama3.2:3b, qwen2.5:3b, qwen2.5:7b, llama3.1:8b), protocol: temp=0, seed=42, context 8192, up to 10 agent steps, three repetitions per (model×approach×task) cell; 609 completed cells recorded; scoring via deterministic rule-based checks with fuzzy matching for free-text.
Themesproductivity org_design
GeneralizabilityEvaluated on the Sakila sample database — a small, well-structured demo schema that may not reflect complexity of real enterprise schemas (many tables, custom business rules, denormalized data)., Task suite is curated and focused on common retrieval patterns; results may not hold for open-ended analytics, ad-hoc exploratory queries, or unfamiliar domains., Models tested are local instruction-tuned models (3B–8B); results may differ for other architectures, very large frontier models, or cloud API models with different prompting/tool-calling behavior., Benchmark measures correctness, latency and token cost but not downstream economic outcomes (e.g., developer productivity, headcount effects, time-to-deploy), so mapping to economic impact requires additional study., Assumes availability of domain engineers to author and maintain verticalized packs; operational costs of creating/maintaining packs are not fully quantified.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The verticalized domain-tool pack achieved a pooled mean accuracy of 0.939, compared with 0.666 for raw SQL and 0.605 for the generic thin-tool pack. Decision Quality | positive | Pooled task accuracy |
Reading fidelity
high
Study strength
medium
|
n=609
0.939 pooled mean accuracy versus 0.666 for raw SQL and 0.605 for the generic pack
|
| The verticalized pack had a higher proportion of fully correct cells than both raw SQL and the generic thin-tool pack: 85% versus 33% and 31%, respectively. Decision Quality | positive | Share of benchmark cells passing all checks |
Reading fidelity
high
Study strength
medium
|
n=609
174/204 (85%) fully-correct cells versus 67/201 (33%) for raw SQL and 63/204 (31%) for the generic pack
|
| The verticalized pack improved accuracy for every tested model, with the largest improvement for the 3B llama3.2 model, whose score increased from 0.583 with raw SQL to 0.929 with domain tools. Decision Quality | positive | Model-level benchmark accuracy |
Reading fidelity
high
Study strength
medium
|
n=4
llama3.2:3b: 0.583 → 0.929
|
| The verticalized pack reduced tokens per correct answer relative to raw SQL for every model, with reductions ranging from 2.0× to 11.6×. Organizational Efficiency | positive | Inference tokens consumed per correct answer |
Reading fidelity
high
Study strength
medium
|
n=4
2.0×–11.6× reduction in tokens per correct answer
|
| The verticalized pack was both more accurate and faster than the other two designs in the benchmark, with mean latency of 4.4 seconds per cell versus 6.4 seconds for the generic pack and 17.3 seconds for raw SQL. Task Completion Time | positive | Mean latency per benchmark cell |
Reading fidelity
high
Study strength
medium
|
n=609
4.4 s per cell versus 6.4 s and 17.3 s
|
| The generic thin-tool pack underperformed raw SQL on pooled mean accuracy, despite exposing parameterized tools: 0.605 versus 0.666. Decision Quality | negative | Pooled task accuracy |
Reading fidelity
high
Study strength
medium
|
n=609
0.605 versus 0.666
|
| Verticalized tools produced the largest accuracy gains on tasks involving recommendation with negative filtering or multi-tool workflows, with improvements over raw SQL ranging from 41.6 to 50.0 percentage points. Decision Quality | positive | Per-task mean benchmark score |
Reading fidelity
high
Study strength
medium
|
n=17
+41.6pp to +50.0pp
|
| For simple single-query tasks such as not_found and recommend_rating, all three approaches achieved a mean score of 1.000, while the empty-result task rental_empty scored 0.667 for every approach. Decision Quality | null_result | Per-task mean benchmark score |
Reading fidelity
high
Study strength
medium
|
n=17
1.000 for not_found and recommend_rating; 0.667 for rental_empty across all approaches
|