Execution-feedback refinement is the only low-cost, universally helpful module in modern in-context learning text-to-SQL pipelines, while most other components help only for certain models; under fixed budgets, building richer pipelines on mid-tier models frequently outperforms buying access to the latest frontier model.

Main Finding

Execution-feedback refinement (run-and-rewrite only when execution fails) is the single modular paradigm whose marginal benefit is robust across diverse LLM backbones while incurring consistently low cost. Most other commonly used modules (various retrieval, schema-linking, generation-strategies, and selection paradigms) give benefits that depend strongly on the backbone’s capability and generation style. Under a fixed budget, it is often more cost-effective to invest in a more elaborate inference pipeline on a mid-tier backbone than to upgrade to a frontier model with a minimal pipeline.

Key Points

• Scope and decomposition

  • The ICL text-to-SQL pipeline is decomposed into three stages (Pre-processing, SQL generation, Post-processing) and five recurring modules:
  • Example Retrieval (E1: question similarity; E2: masked-question; E3: SQL-skeleton)
  • Schema Linking (S1: retrieval-based; S2: pseudo-SQL guided; S3a/b: direct LLM linking one-shot/two-stage; S4: agentic)
  • Generation Strategy (G1: direct; G2: chain-of-thought enhanced; G3: decomposition)
  • Candidate Selection (C1: execution-based voting; C2a: scoring/ranking; C2b: pairwise)
  • Refinement (R1: execution-feedback; R2: self-correction w/o exec; R3: agentic refinement)
  • 17 paradigm-level configurations instantiated (24 representative methods from the literature mapped to these paradigms).

• Experimental setup

  • Primary dataset: BIRD (dev, 1,534 examples); transfer check on Spider (1,034).
  • Primary backbones: GPT-4o-mini, Gemini-2.5-Flash, DeepSeek-V4-Flash (reasoning-oriented), GPT-5.4. Additional checks with Qwen3-235B (thinking mode) and o3-mini; guideline validated on five extra backbones.
  • Controlled, single-swap protocol: replace only one module at a time in a unified implementation so marginal contribution of that paradigm can be attributed reliably.
  • Baselines: trivial settings (no retrieval, full schema, G1 direct gen, N=1 except matched N=5 for candidate-selection).

• Metrics and cost accounting

  • Primary accuracy metric: execution accuracy (EX). Oracle candidate recall (OCR) reported for multi-candidate setups.
  • Cost-aware metric: CPP (cost per EX point) — incremental USD required to gain one percentage point of EX over the matched reference (reported only when EX increases and cost increases).
  • Token accounting: mean per-example input (IN) and output (OUT) tokens tracked to characterize resource profiles beyond aggregate API cost.

• Principal empirical findings

  • Universal win: R1 (execution-feedback refinement) improves EX consistently across all tested backbones at low incremental cost — making it a safe, default pipeline addition.
  • Backbone dependence: most other paradigms show benefits only for some backbones and can be costly (high CPP) or even neutral/harmful on others. Aggregate end-to-end accuracy masks these conditional effects.
  • Candidate selection: when multiple samples are already drawn (N=5), execution-based voting (C1) can yield statistically significant EX gains at no extra cost beyond the sampling budget (i.e., good value if you already sample).
  • Token dynamics: input token demand is more closely tied to pipeline structure (e.g., retrieval and schema linking increase prompt length), while output token demand reflects backbone generation behavior (different LLMs produce more/less verbose outputs).
  • Stacking: adding modules (stacking) generally increases EX but with diminishing returns; how gains compose depends on backbone capability and reasoning style.
  • Budget allocation: in many cases a mid-tier backbone plus a richer pipeline outperforms a frontier backbone with a lean pipeline for the same dollar budget.
  • Transfer: the authors distilled guidelines into a tiered, cost-efficient pipeline prescription and validated it on five additional backbones without per-paradigm search.

Data & Methods

• Datasets: BIRD (main, dev split), Spider (transfer evaluation).
• Backbones: GPT-4o-mini, Gemini-2.5-Flash, DeepSeek-V4-Flash, GPT-5.4; additional checks as above.
• Controlled implementation:
  • Unified framework re-implements each paradigm so differences reflect paradigms, not ad-hoc system choices.
  • Single-swap protocol: swap exactly one module while keeping others trivial.
  • Matched baselines: use N=1 as baseline except when evaluating selection (matched N=5).
• Metrics:
  • EX (execution accuracy), OCR (oracle recall across N candidates).
  • Cost per EX point (CPP) computed as incremental USD divided by incremental EX percentage points relative to matched reference.
  • Input/output token counts per example reported.
• Statistical tests: McNemar’s test for EX improvements, 95% CIs reported.

Implications for AI Economics

• Marginal-cost-aware design matters: reporting only end-to-end accuracy is insufficient for practical deployment decisions. Practitioners and procurement teams should evaluate marginal accuracy per-dollar (e.g., CPP) when choosing between (a) richer inference pipelines on cheaper models and (b) more expensive backbone upgrades.
• Prioritization rule (actionable):
  • First add execution-feedback refinement (R1): low-cost, robust accuracy improvement across backbones.
  • If sampling multiple candidates already, use execution-based voting (C1) — it often yields free gains within the same sampling budget.
  • Beyond those, evaluate module-by-module on your chosen backbone: gains from retrieval, schema linking, CoT/decomposition, or agentic refinement are backbone-dependent; run small controlled swaps to measure CPP before scaling.
• Procurement and deployment strategy:
  • For constrained budgets, prefer investing in pipeline engineering around a mid-tier model rather than immediately upgrading to a frontier model; this often provides a better EX-per-dollar tradeoff.
  • Token accounting (IN vs OUT) should be tracked alongside dollars because architectures that lengthen prompts increase input tokens (and prompt costs) predictably, while output verbosity depends on the model and can materially change costs.
• Research and reporting norms:
  • Authors should publish marginal, module-level cost/accuracy analyses (or at least CPP-style metrics) in addition to aggregate end-to-end scores so consumers can make informed engineering/economic trade-offs.
  • Standardized controlled-implementation protocols (like the single-swap protocol here) help disentangle where gains originate — useful for reproducible, economically grounded evaluation.

Caveats and open directions - Results are grounded on BIRD (dev) and a controlled implementation; real production datasets, larger schema heterogeneity, or other LLM pricing/latency regimes could change the exact CPPs. - Costing assumes particular API/token prices and does not include engineering or latency costs of building complex agentic systems. - Future work: automated budget-constrained pipeline-search, evaluation across more benchmarks and realistic deployment constraints (latency, privacy, fine-tuning costs).