Hiding lightweight online learners inside LLM round-trips lets a semantic-SQL engine cut expensive LLM calls roughly eightfold in realistic workloads (11.4× under independence). The production Cortex AISQL case study shows per-call selectivity learning and per-batch calibrated routing compose multiplicatively to substantially lower token and latency costs.

Main Finding

Compositional online learning at the LLM-call boundary—i.e., running multiple CPU-side online learners whose updates are hidden inside the LLM round-trip—can substantially reduce LLM-dominated query cost in semantic-data processing systems. When instantiated in a production system (Cortex AISQL) as three components (response caching, per-call filter-ordering (Larch), and per-batch cascade-routing (GAMCAL)), the framework yields a theoretical multiplicative cost reduction of up to 11.4× on a representative conjunction-filter workload under independence; realistic effects after cross-component interactions put that figure near 8×. The framework also changes optimizer design constraints by exploiting the long LLM latency to afford heavier online learners (per-call gradient steps and per-batch threshold solves) that would be infeasible at relational latencies.

Key Points

• Cost context

  • In production AISQL, LLM invocation dominates query cost (≈80–90%).
  • A single LLM call is roughly 10^5–10^7× more expensive than a relational predicate, so reducing LLM calls or delegations yields large economic impact.
  • End-to-end semantic queries can have latencies measured in hours due to many LLM calls.

• The LLM-bound design point

  • Each LLM round-trip (hundreds of milliseconds) is wide enough to hide non-trivial CPU-side training/update steps.
  • This inverts classical AQP constraints: heavier per-call or per-batch learners become affordable because their compute is absorbed by LLM latency.

• Framework (composition at the LLM boundary)

  • Design axes: decision granularity (per-call, per-row, per-batch) × learner update cadence (online per-call, online per-batch, one-shot, none/memoization).
  • Shared operational pattern: predict → issue LLM call while a background trainer consumes buffered supervision and writes back updated artifacts; trainers are typically one round (or one batch) behind but do not block LLM calls.
  • Components can be composed; under independence they decompose cost into multiplicative factors that can be optimized separately.

• AISQL instantiation (case study)

  • Components:
    • Response caching (memoization; per-call, no learner): suppresses calls on exact matches.
    • Larch (filter ordering; per-call online): refits a per-predicate selectivity model after every LLM outcome and reorders remaining predicates per row to exploit short-circuiting.
    • GAMCAL (cascade routing; per-batch online): refits a calibrated GAM on proxy scores and solves thresholds to route uncertain rows to the oracle LLM.
  • Composition result: conditional cost decomposition assigns each learning component to a distinct factor of per-row LLM cost; multiplicative upper bound 11.4×; realistic ~8× after accounting for cross-component interactions (self-selection at cascade boundary, sample-budget shrinkage, selectivity-estimation drift).

• Empirical performance highlights

  • Larch-Sel (selectivity MLP + DP solver) reduces token-overhead dramatically vs prior offline/compile-time estimators (Palimpzest, Quest): typical reductions in 4–8× band; per-dataset token-overhead relative to per-row optimal lower bound: GovReport 5.1%, PubMed 6.5%, BigPatent 3.6% (Larch-Sel) vs 21–33% for baselines.
  • Larch training steps take ≈7–11 ms (CPU) and fit inside typical LLM round-trips (hundreds of ms).
  • DP solver for order selection runs in ~20 ms for n=10 predicates (O(n·3^n) ops).

• Important caveats

  • Multiplicative composition relies on independence between the cost factors targeted by components; real workloads introduce correlations that reduce idealized gains.
  • Three concrete cross-component interactions reduce theoretical gains to realistic ones: self-selection at cascade boundary (interaction between ordering and routing), sample-budget shrinkage for GAM calibration, and selectivity-estimation drift over time.
  • The background training is one-round stale; AISQL ablation shows this staleness produces negligible token-cost change (±0.6% average).

Data & Methods

• Substrate and workloads

  • Production substrate: Cortex AISQL (native semantic operators embedded in SQL), with AI FILTER, AI CLASSIFY, AI JOIN, AI COMPLETE, etc.
  • Representative workload: Mix (50% AND, 50% OR) conjunction-filter workloads; datasets include GovReport, PubMed, BigPatent (used for Larch evaluation).

• Filter ordering (Larch-Sel)

  • Learner: shared two-layer MLP (≈144K parameters) outputting per-predicate pass probability (sigmoid), trained with binary cross-entropy.
  • Features: pre-computed document and predicate embeddings (ingestion-time), projected to 64-D via learned linear maps; concatenation includes d, f, d⊙f, cos(d,f).
  • Planner: exact DP solver for AND/OR trees using recurrence OPT(T′) (solvable exactly under independence); solver complexity O(n·3^n).
  • Update cadence: online per-call—one BCE gradient step per LLM outcome, run in background and written back while next LLM call completes.

• Cascade routing (GAMCAL)

  • Proxy + oracle pattern: cheap proxy score per row; oracle is the expensive LLM.
  • Learner: monotone smoothing-spline calibrated GAM fit by penalized maximum likelihood with smoothness penalty (and monotonicity constraint); fit on buffered oracle labels.
  • Routing rule: two thresholds (τlow, τhigh) chosen to minimize a cost-quality objective that trades F1 degradation vs delegation rate with user knob α.
  • Stochasticized calibrated score: add Gaussian-drawn quantile based on per-score posterior se to encourage exploration.
  • Update cadence: online per-batch—refits (calibrator and thresholds) on a doubling schedule; each refit hides inside the aggregate latency of the batch’s oracle calls.
  • Sampling: each batch randomly samples rows from the uncertain region to add labels to the buffer; first refit requires minimum labels per class.

• Composition theory & analysis

  • Formalized a sequential composition rule and a conditional cost decomposition that assigns distinct per-row LLM cost factors to the learning components (e.g., expected evaluated predicates per row ¯e, delegation fraction pdeleg).
  • Under independence, components’ savings multiply; analytic case study yields 11.4× upper bound.
  • Cataloged three cross-component interactions that reduce realized gains, and measured realistic effects (~8×) on a representative workload.

Implications for AI Economics

• Large leverage from optimizer-side investment

  • Because LLM compute dominates cost, relatively inexpensive CPU-side investments (models, solvers, caching, orchestration) can yield outsized reductions in billed LLM usage and thus large cost savings. The reported ~8× realistic reduction indicates major ROI potential for engineering and research effort on online learners and compositional optimizers.

• Rethink of optimizer design and resource allocation

  • The LLM-latency window changes design trade-offs: heavier online learners (per-call gradients, per-batch constrained solves) are economically justified and technically feasible. System designers should reallocate some budget from LLM consumption to CPU-side modeling and infrastructure to reduce total LLM calls.

• Pricing, provisioning, and SLAs

  • Adoption of compositional online learning shifts the marginal cost profile of queries: fewer LLM invocations → lower variable costs and different latency/cost trade-offs. Cloud billing and capacity planning should account for:
    • reduced peak LLM throughput needs,
    • increased CPU demand for background training steps (but small relative to LLM cost),
    • the stochasticity of delegation policies that trade accuracy for delegation (alpha knob).
  • SLA design can exploit tunable knobs (e.g., α) to allow customers to trade cost vs quality transparently.

• Multiplicative composition and strategic targeting

  • The conditional cost decomposition suggests targeting orthogonal cost factors (e.g., short-circuiting vs delegation fraction vs caching) yields multiplicative benefits. AI-economics work should prioritize components that act on independent factors first to maximize combined savings.
  • However, independence is a strong assumption; practical deployments must measure cross-component correlations and factor in diminishing returns and interactions.

• Implications for model suppliers and marketplaces

  • Systems that can reliably reduce LLM calls will shift demand away from raw token usage toward predictor/caching infrastructure. This could change market dynamics (e.g., lower recurring LLM spend, increased spending on embedding/cost-optimization tooling).
  • Vendors of LLM inference might see pressure to offer cheaper fine-grained invocation tiers or better primitives (e.g., cheaper small models for proxies) that integrate with cascade routing.

• Environmental and unit-cost externalities

  • Substantially reduced LLM invocation rates reduce energy consumption and carbon footprint per query; this has macroeconomic and regulatory relevance as LLM usage scales.

• Future research & investment directions (economic priorities)

  • Invest in: robust compositional cost models that measure cross-component interactions; per-call selectivity models that are sample-efficient; reliable cascade calibration under scarce labeling budgets.
  • Explore: prompt caching, model substitution (distillation into cheaper student nets), adaptive batching, and prompt-structure learning as additional cost-reduction levers, all evaluated by marginal cost saved per engineering/compute dollar spent.

Summary: In LLM-dominated semantic query processing, online learners whose updates are hidden within LLM round-trips unlock a new, high-leverage design space. Composing orthogonal learning components can multiplicatively reduce LLM consumption—and thus monetary cost—by factors measured in single-digit to low-double-digit multiples in realistic settings. For AI-economics, this points to high returns from investing in optimizer-side modeling, careful composition analysis, and systems that expose configurable cost/quality trade-offs.