Confine the model, not the math: structural abstention prevents LLMs from inventing numbers by letting generative models shape and confirm questions but delegating all value computation to a deterministic kernel; a two-year enterprise deployment shows the approach is practicable though it narrows coverage.

Main Finding

Large language models (LLMs) driving natural-language-to-database interfaces are unreliable when their generated queries are invisible to the consumer: fluent but fabricated or mis-aggregated numbers arrive as undetectable facts. The paper proposes an architectural pattern—structural abstention—centred on a perimeter invariant: any component that can fabricate may influence which question the system answers (phrasing, clarification) but must never influence the value returned (the number or executed action). Concretely, a generative "shell" handles interpretation and interaction up to an explicit confirmation of a fully specified question; a deterministic, non-generative "kernel" (grounder + compiler + executor) takes only that confirmed question and computes the value. Unanswerable or out-of-scope requests are declined before any value is produced, making refusal a first-class, observable outcome rather than a probabilistic after-the-fact abstention.

Key Points

• The perimeter invariant: generative components may shape/phrase questions but must never produce values that flow to consumers as facts. Everything that could fabricate is kept outside the correctness-critical path.

• Shell vs Kernel:

  • Shell (may be generative): INTERPRETER (extract determinate content), ELICITOR (ask clarifying questions, phrase confirmations and results).
  • Kernel (must be non-generative/deterministic): GROUNDER (decidable test: either returns a grounded question or reports failure), COMPILER (pure function mapping grounded question → query), EXECUTOR (runs query against data store).

• Structural abstention vs statistical abstention:

  • Structural abstention: refuse at grounding time by making unanswerable requests unrepresentable—no confidence estimate required.
  • Statistical abstention: generate candidate answer then use model-based uncertainty to decide to abstain; requires calibration and is still vulnerable to undetected fabricated values.

• Dialogue policy (forced by GROUNDER totality): three exhaustive branches

• If state inconsistent → prompt user to resolve conflict.
• If GROUNDER grounds → show confirmation; on user assent, deterministically compute and render result.

• Else if incomplete → elicit missing pieces; otherwise, disclose nearest answerable questions (make capability boundary explicit).

• Guarantees from the pattern:

• No fabricated returned values; all numeric/factual outputs come from deterministic execution.
• Auditability without logging: confirmed question uniquely determines the query (COMPILER is a function).

• Failures are visible refusals (coverage errors), not silent incorrect answers.

• Trade-off: reduced coverage and flexibility—the system declines some user requests it could have approximated—trading breadth for auditable reliability. The paper identifies conditions where that trade-off is worthwhile (noted qualitatively).

• Extension to agents: invariant generalized from "never the number" to "never the executed action": generative parts may propose actions but must not control what action actually executes.

• Empirical / practical context:

  • Production two-year case study (enterprise analytics) implemented this pattern; specifics are redacted but used to compare against two generative alternatives (fine-tuned parser and a tool-retrieval / RAG agent). RAG/tool-agent prototype improved coverage but incurred latency and nondeterminism; deterministic kernel gave auditable reliability necessary for fact-consumed settings.
  • Recent benchmarks (Spider 2.0, BEAVER) show sharp drops in off-the-shelf model performance on enterprise-style schemas—motivating constrained, auditable approaches.

Data & Methods

• Nature of work: architecture and design paper with an industry production case study; no public datasets, code, or private schema details are disclosed. Examples and a synthetic schema are used for illustration.

• Methodological components:

  • Pattern specified implementation-independently, then reduced to a five-decision recipe (pattern → instantiation choices) and applied across multiple domains (enterprise analytics, clinical ward reporting, financial reporting).
  • Roles and obligations formalized (INTERPRETER, ELICITOR, GROUNDER, COMPILER) with explicit constraints: GROUNDER must be total (return question or explicit failure); COMPILER must be a deterministic function.
  • Dialogue-state representation that encodes partial specification explicitly and supports three operations: completeness test, gap enumeration, consistency test.
  • Deterministic compilation/execution path (kernel) replaces any generative generation of query or value.
  • Comparison experiments in production against two generative alternatives:
    • Fine-tuned parser (end-to-end text-to-SQL).
    • Retrieval-augmented-generation (tool-retrieval agent) that calls helpers and composes queries (improved legibility but added nondeterminism/latency).
  • Component engineering choices mentioned: attention-based entity relevance, DSSM-style dual encoder for question recommendation, inference acceleration, compact translation models, neural NER—used within the shell or ancillary tooling but not inside the kernel.

• Limitations of methods: no quantitative public metrics provided; evaluation is qualitative and based on production experience plus examination of external benchmarks that measure the specific failure modes the pattern targets.

Implications for AI Economics

• Reliability vs Coverage trade-offs change product-market fit:

  • For contexts where consumers cannot inspect queries (enterprise dashboards, automated agents acting on values, clinical/financial decisions), the economic cost of a single undetected erroneous value can far exceed gains from broader coverage. Structural abstention is economically justified when downstream error costs are high and audits are infeasible.
  • Products that prioritize auditable, deterministic outputs will be preferable for high-stakes applications, even if they answer fewer queries automatically.

• Adoption and labor implications:

  • The pattern institutionalizes a hybrid human–AI workflow: automated handling of auditable queries + explicit refusals that route to human analysts. That preserves human oversight where needed and may reshape labor demand toward more complex analytical tasks rather than routine query writing.
  • Firms may adopt such constrained systems to reduce legal/operational risk, affecting procurement and investment decisions (preference for certified determinism over best-effort LLM outputs).

• Pricing, contracts, and liability:

  • Systems using structural abstention create clearer contractual guarantees (e.g., auditable computation path, explicit refusal behavior), enabling different pricing tiers and liability allocations compared to black-box LLM services.
  • Insurers and regulators may favor/require deterministic kernels for mission-critical fact delivery, influencing market standards and certification regimes.

• Innovation direction and R&D allocation:

  • Benchmarks should shift to measure not only average execution accuracy but the reliability of refusal and auditable determinism (coverage vs safe-decline trade-offs). Investment may move from pushing raw average accuracy to improving deterministic grounding, schema abstraction layers, and coverage expansion techniques that retain kernel determinism (e.g., deterministic resolvers, verified tool-chains).
  • Agentic AI research must prioritize invariants that separate proposal (generative) from execution (deterministic) to maintain safety in automated action settings.

• Competitive strategy and product design:

  • Companies targeting enterprise analytics should evaluate whether to invest in constrained semantic layers and deterministic compilers (higher upfront engineering cost, lower downstream risk) versus offering broader LLM-based coverage (faster feature development but higher exposure to silent errors).
  • Structural abstention can be a differentiator in regulated industries (healthcare, finance), enabling premium positioning for trustworthy automation.

• Macroeconomic effects:

  • By enabling safer automation in settings where numbers drive automated decisions, structural abstention can accelerate deployment of AI-driven operational improvements while limiting costly error-induced disruptions—raising productive adoption in sectors previously cautious about LLMs.

In short: the paper reframes reliability as an architectural constraint (keep the fabricator out of the value path) and shows how making refusal explicit and deterministic yields auditable, safer interfaces for fact-consumed AI applications. The economic implications favour constrained, certifiable designs when the cost of silent errors is high and suggest new product, regulatory, and benchmarking directions.