0 cumulative citations
View corpus contextAn LLM-driven query engine, GenDB, generates instance-optimized execution code that materially outperforms conventional engines on TPC-H and a custom benchmark for repeated templated queries; the gains rely on amortizing upfront code-generation costs and on hardware- and data-specific optimizations, limiting applicability to repeatable workloads.
Demonstrating GenDB: Instance-Optimized and Customized Query Processing Code Generation via LLM Agents
GenDB is a prototype LLM-driven query engine that generates instance- and hardware-optimized query execution code and achieves substantially better performance than state-of-the-art engines on TPC-H and a bespoke benchmark for repetitive templated queries, with correctness validated via fuzz testing and manual inspection.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Traditional query processing engines require continuous development and extensions to support new techniques and user requirements, and in some cases, entirely new systems must be built from scratch. However, these engines are difficult to extend due to their internal complexity, and building new systems demands significant engineering effort and cost. To address this, we demonstrate GenDB, a generative query engine that shifts query processing from manually engineered systems to query processing code generation driven by Large Language Models (LLMs). An early prototype of GenDB uses LLM agents to generate instance-optimized query execution code tailored to specific data, workloads, and hardware resources. This prototype suits offline code generation for repetitive, templated queries, since the upfront generation cost amortizes over many executions and correctness can be ensured through extensive fuzz testing and manual inspection. For ad-hoc queries, GenDB can work with a traditional DBMS in a hybrid architecture: the DBMS handles one-off queries, while GenDB speeds up frequent SQL templates. Our demonstration allows users to (1) visually and interactively explore how GenDB analyzes workloads, profiles hardware resources and underlying data, produces query plans, generates code based on them, and finally uses an optimizer to iteratively achieve a correct and efficient implementation; (2) use visual inspection and analysis to gain qualitative insights into why GenDB produces code that achieves significantly better performance than state-of-the-art query engines on two benchmarks: TPC-H and a newly constructed benchmark designed to reduce potential data leakage from LLM training data; and (3) upload their own data and queries to explore GenDB with different LLMs and query patterns.
Summary
Main Finding
GenDB is a prototype LLM-agent–driven system that generates instance-optimized, hardware-aware, per-template query-execution code (and corresponding storage/index layouts). For repetitive, templated queries, GenDB’s offline code generation plus iterative, runtime-driven refinement yields large speedups over state-of-the-art DB engines (e.g., 2.4–9.2× on TPC-H and 3.3–3.8× on a held-out SEC-EDGAR benchmark in the authors’ experiments), while correctness is validated empirically (fuzz testing and cross-checks with a traditional DBMS).
Key Points
- Approach
- Decomposes end-to-end query processing into LLM agents: Workload Analyzer, Storage/Index Designer, Query Planner, Code Generator, Query Optimizer.
- Agents use tools (file ops, terminal, web search) and LLM reasoning to design storage, pick join orders, generate C++ code for each SQL template, and iteratively refine via measured feedback.
- Target use case: offline generation for repetitive, templated queries common in industrial workloads (upfront cost amortized over many runs).
- Performance
- Demonstrated speedups vs DuckDB, Umbra, ClickHouse, MonetDB, PostgreSQL on TPC-H (SF=10) and a new SEC-EDGAR benchmark designed to avoid LLM training-data leakage.
- Examples: total TPC-H time 249 ms (GenDB) vs 596 ms (DuckDB) and 660 ms (Umbra); some queries improved 22× through iterative fixes.
- Correctness & Validation
- Validated by comparing outputs to a traditional DBMS and fuzz testing across template predicates and test DBs.
- Authors acknowledge empirical validation ≠ formal semantic equivalence; future work planned on stronger guarantees (e.g., formal verification).
- Costs & Practicalities
- Prototype uses Claude Sonnet 4.6; synthesis cost/time reported: ~$14 & 91 minutes (TPC-H), ~$23 & 140 minutes (SEC-EDGAR).
- Generation cost is considered acceptable for high-repeat workloads; authors propose lowering costs by using smaller models for simple steps and reusable operators.
- Limitations
- High generation latency and monetary cost (today).
- Lack of formal correctness guarantees.
- Best suited for recurring, templated queries; not a drop-in replacement for ad-hoc query processing (hybrid architectures suggested).
- Assumes ability to run or compare against a ground-truth DBMS for validation and often assumes hot cached datasets for reported runtimes.
Data & Methods
- Prototype & stack
- Implementation: JavaScript prototype; Claude Agent for multi-agent orchestration; C++ as target for generated high-performance executables.
- Agent tooling: file/terminal/web access to enable sampling, profiling, code edit/run cycles.
- Evaluation environment
- Hardware: server with 2× Intel Xeon Gold 5218, 384 GB RAM; databases cached in memory (hot-run timings).
- LLM: Claude Sonnet 4.6 used in experiments (costs/time reported).
- Benchmarks & baselines
- TPC-H scale factor 10 (well-known, likely in LLM training data).
- SEC-EDGAR: newly constructed benchmark (SQLSmith-generated queries over a dataset rarely used for DB benchmarking) to reduce potential LLM-data leakage.
- Baselines: DuckDB, ClickHouse, Umbra, MonetDB, PostgreSQL; experiments used each baseline’s best configuration and optionally supplemented with indexes recommended by GenDB.
- Evaluation procedure
- Per-template generation of executables; iterative optimization loop: generate → run → profile → refine until budget or target reached.
- Correctness checks via comparison to traditional DBMS outputs and fuzz testing across parameterizations and test databases.
- Reported metrics: per-query and total runtime, iteration-level improvements, generation cost (time + $).
- Reproducibility
- Artifacts available: code, data, and demos published at https://github.com/SolidLao/GenDB and an interactive demo.
Implications for AI Economics
- Labor substitution and productivity gains
- GenDB illustrates automation of traditionally expert-heavy engineering (low-level engine extensions, hand-tuned C++ per-query code). If matured, it could reduce demand for highly specialized DB-engine engineering time and raise productivity for system deployment and tuning.
- New value-capture models & pricing implications
- Vendors or cloud providers could monetize generation-as-a-service: one-time/periodic code-generation fees for recurring workloads (pricing tied to model compute/time) with payback via reduced per-query execution cost.
- Economics favor workloads with high repetition: break-even when generation cost is less than cumulative runtime savings across expected executions.
- Operational cost trade-offs
- Upfront LLM generation costs (compute and monetary) vs ongoing execution savings (CPU, memory, I/O, latency). Customers will need simple ROI/break-even analyses (generation cost ÷ expected runs = amortized cost per run).
- Model inference (LLM) costs, latency, and carbon footprint become part of total cost of ownership for database systems.
- Market disruption and competitive dynamics
- If effective, automated instance-specific engines could commoditize aspects of query-engine differentiation (vendors may compete on generator quality, correctness guarantees, and integration rather than raw engine internals).
- Hybrid strategies emerge: traditional engines for ad-hoc queries + gen-engine for hot templates—this can reshape product bundling and SLAs.
- New services and third-party markets
- Opportunities for third-party providers offering optimized generator pipelines, verification/certification services (formal verification, testing suites), or smaller specialized LLMs/agents for low-cost generation.
- Risk, liability, and trust economics
- Incorrect optimizations carry risk (wrong results, outages). Economic value depends on verifiable correctness; demand for formal guarantees or insurance/SLAs will increase.
- Data leakage and regulatory compliance concerns (LLMs trained on proprietary data) may affect adoption where data privacy/regulatory constraints are tight.
- Strategic implications for firms
- Large cloud/analytics customers with recurring workloads stand to gain most; smaller or ad-hoc-query-heavy users may not benefit unless generation costs drop.
- Firms should evaluate portfolio-level decisions: which query templates to synthesize, expected lifetime runs, and required verification overhead.
- Research & investment priorities
- Near-term returns: invest in tooling that reduces LLM cost (model distillation, pipeline decomposition), verification tech, and reusable operator libraries to amplify ROI.
- Long-term: standards for correctness guarantees and benchmarking for LLM-generated systems will be economically important for market trust.
Actionable suggestion for practitioners: perform a simple break-even calculation before adopting GenDB-like workflows: amortized generation cost per run = generation_cost / expected_number_of_executions. Compare this to per-run cost savings on your cloud/compute bill (from faster runtimes or reduced resource usage) and to added verification/maintenance overhead.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents empirical performance comparisons on established (TPC-H) and a bespoke benchmark and supplies qualitative analysis and fuzz testing to support correctness, but it lacks randomized or quasi-experimental identification, formal statistical inference, and independent replication; benchmark selection and prototype tuning could explain some gains.
Methods Rigormedium — The prototype evaluation uses standard benchmarks, hardware profiling, and fuzz testing plus manual inspection, which is appropriate for a systems demonstration, but the study does not present comprehensive ablation tests, statistical uncertainty, sensitivity analyses across diverse workloads/hardware/LLMs, or checks against overfitting to the chosen benchmarks.
SampleExperimental evaluation on the TPC-H benchmark and a newly constructed benchmark intended to avoid LLM training-data leakage; tests compare runtime and resource use of LLM-generated, instance-optimized query execution code (GenDB) against state-of-the-art query engines, with profiling across specific hardware configurations and templated query workloads; also supports user-supplied data/queries for interactive demos.
Themesproductivity innovation human_ai_collab
GeneralizabilityOptimization gains demonstrated primarily for repetitive, templated queries where code generation costs can be amortized; not validated for ad-hoc queries except in a proposed hybrid architecture, Performance improvements may be hardware-specific because generated code is instance- and hardware-optimized, Results may depend heavily on the particular LLM(s) used and their training/coverage; benchmark construction and selection risk favorable tuning, Correctness assurances rely on fuzz testing and manual inspection, which may miss rare edge cases in production-scale data, Scalability and maintenance costs of generated code over evolving schemas/workloads are not fully evaluated, Security, robustness to adversarial inputs, and operational engineering costs are not addressed
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Traditional query processing engines require continuous development and extensions to support new techniques and user requirements, and in some cases, entirely new systems must be built from scratch. Developer Productivity | negative | need for continuous development and system replacement |
Reading fidelity
high
Study strength
low
|
not reported
|
| These engines are difficult to extend due to their internal complexity, and building new systems demands significant engineering effort and cost. Developer Productivity | negative | engineering effort and cost required to extend or build DBMSs |
Reading fidelity
high
Study strength
low
|
not reported
|
| We demonstrate GenDB, a generative query engine that shifts query processing from manually engineered systems to query processing code generation driven by Large Language Models (LLMs). Innovation Output | positive | approach to query processing (LLM-driven code generation vs. traditional engineered systems) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| An early prototype of GenDB uses LLM agents to generate instance-optimized query execution code tailored to specific data, workloads, and hardware resources. Task Completion Time | positive | generation of instance-optimized query execution code |
Reading fidelity
high
Study strength
medium
|
not reported
|
| This prototype suits offline code generation for repetitive, templated queries, since the upfront generation cost amortizes over many executions. Task Completion Time | positive | suitability for offline code generation on repetitive/templated queries (amortized cost) |
Reading fidelity
high
Study strength
low
|
not reported
|
| Correctness of generated code can be ensured through extensive fuzz testing and manual inspection. Output Quality | positive | correctness / reliability of generated query execution code |
Reading fidelity
high
Study strength
medium
|
not reported
|
| For ad-hoc queries, GenDB can work with a traditional DBMS in a hybrid architecture: the DBMS handles one-off queries, while GenDB speeds up frequent SQL templates. Task Allocation | positive | division of query handling between DBMS (one-off) and GenDB (frequent templates) |
Reading fidelity
high
Study strength
low
|
not reported
|
| GenDB produces code that achieves significantly better performance than state-of-the-art query engines on two benchmarks: TPC-H and a newly constructed benchmark designed to reduce potential data leakage from LLM training data. Task Completion Time | positive | query execution performance on benchmark workloads |
Reading fidelity
high
Study strength
medium
|
n=2
|
| The demonstration allows users to visually and interactively explore how GenDB analyzes workloads, profiles hardware resources and underlying data, produces query plans, generates code based on them, and finally uses an optimizer to iteratively achieve a correct and efficient implementation. Developer Productivity | positive | interactive exploration and visualization of GenDB's end-to-end workflow |
Reading fidelity
high
Study strength
low
|
not reported
|
| Users can upload their own data and queries to explore GenDB with different LLMs and query patterns. Adoption Rate | positive | ability for users to run custom data/queries and choose different LLMs |
Reading fidelity
high
Study strength
low
|
not reported
|