An LLM-powered tool, RepGen, reproduces 80% of 106 real-world deep-learning bugs—about 20 percentage points better than the prior best—while a 27-developer study shows it raises success rates by 23% and halves debugging time, implying meaningful productivity gains for ML engineers.

Main Finding

RepGen — an LLM-powered, agentic system that builds a learning-enhanced project context, generates a targeted reproduction plan, and runs an iterative generate–validate–refine loop — reproducibly generates runnable code that reproduces real-world deep learning (DL) bugs. On a curated benchmark of 106 DL bugs from 16 GitHub projects, RepGen reproduced 85 bugs (80.19%), outperforming LLM-only baselines and improving the prior state-of-the-art measure by ~19.8%. In a controlled developer study (n = 27), RepGen increased developer reproduction success by 23.35%, reduced reproduction time by 56.8%, and lowered cognitive load.

Key Points

• Problem addressed: reproducing DL bugs is hard due to nondeterminism, hardware/framework dependencies, scattered pipeline artifacts, and “silent” bugs (e.g., degraded model loss/accuracy rather than crashes).
• Core idea: combine project-specific retrieval (code + dependencies + training loops) to form a learning-enhanced context, generate a plan for reproduction, and let an LLM-driven agent iteratively generate, validate, and refine reproduction code using multi-source feedback.
• Retrieval & context:
  • Semantic chunking (AST-based) and dual indexing (sparse BM25 + dense embeddings).
  • Hybrid scoring: scoreh(D) = (1 − α)·BM25norm + α·simangular (α tuned to 0.55).
  • Re-rank top K (e.g., 20) with a cross-encoder (ms-marco-MiniLM-L12-v2).
  • Resolve module-level dependencies and extract training loops via AST heuristics (framework-specific patterns like model.fit(), optimizer.step(), etc.).
  • Construct per-module contexts (top ≤5 modules) combining loops, relevant snippets, dependency graphs.
• Reproduction agent:
  • Produces an initial script from context + plan.
  • Validates and refines using structural/static feedback (lint/compilation), relevance feedback, and runtime/symptom checks (including LLM-based evaluation of whether produced behavior matches bug description).
  • Iterative generate–validate–refine until symptoms reproduced or budget exhausted.
• Implementation details & experiments:
  • Evaluated vs. eight LLM-only baselines (GPT-4.1, Llama 3, Qwen variants, DeepSeek-R1, etc.).
  • Ablation study on 13 components shows the contribution of retrieval, loop extraction, re-ranking, and multi-source feedback.
  • Replication package and dataset (106 bugs + artifacts) released for reproducibility.
• Representative results:
  • Overall reproduction: 85/106 = 80.19%.
  • Improvement vs. state-of-the-art measure: +19.81% (absolute).
  • Developer study (27 participants): +23.35% reproduction success; −56.8% time to reproduce; reduced cognitive load reported. Example reproduction completed ≈5 minutes in a motivating case.

Data & Methods

• Dataset: 106 real-world DL bugs collected from 16 GitHub projects; each bug includes bug report text and project artifacts (code, configs, snippets).
• Preprocessing:
  • AST-based semantic chunking to split large files.
  • Indexing: sparse (BM25) and dense (embedding) indices.
• Retrieval pipeline:
  • BM25 for lexical matches (APIs, errors), ANN embedding search for semantic matches.
  • Hybrid score with α = 0.55, selecting top K (e.g., 20) candidate snippets.
  • Cross-encoder-based reranking (ms-marco-MiniLM-L12-v2) to finalize candidates.
• Context construction:
  • Module-centric grouping (top ≤5 modules).
  • AST-driven training-loop extraction using 8 framework-specific heuristics.
  • Dependency resolution (imports, referenced variables) to assemble a runnable context.
• Agentic reproduction workflow:
  • Bug report restructuring via LLM to extract observed vs. intended behavior.
  • Plan generation for environment setup, training, and inference steps.
  • Iterative code generation by LLM (examples used Qwen2.5-Coder-7B among others).
  • Multi-source feedback: static analysis (PyLint/compile errors), relevance feedback (cross-encoder scores), runtime/symptom checks (including LLM-based symptom verification).
• Evaluation:
  • Quantitative reproduction success on 106 bugs.
  • Comparisons against 8 LLM baselines and ablations of system components.
  • Human study (n = 27) comparing developer performance with/without RepGen support.
• Artifacts: replication package with implementation, dataset, and experimental logs available (link in paper).

Implications for AI Economics

• Productivity and labor effects:
  • Direct reduction in developer time to reproduce DL bugs (~56.8% median reduction in the study) implies lower labor costs in ML engineering and faster development cycles. This increases developer throughput and can reduce time-to-market for AI-enabled products.
  • Tools like RepGen act as productivity multipliers (augmentation), shifting developer effort toward higher-level tasks (diagnosis, fixing, validation) rather than environment reconstruction — potentially changing the skill mix demanded in ML teams.
• Cost structure and operational expenditure:
  • Faster, more reliable bug reproduction reduces expensive iterative debugging cycles and lowers the cost of debugging in model development and MLOps. This can reduce operational risk and maintenance costs for deployed AI services.
  • Offsetting costs: running LLM agents, embedding indices, and repeated test runs require compute and possibly paid LLM API usage; organizations must weigh these compute/API costs against saved labor and faster resolution.
• Product reliability, liability, and insurance:
  • Improved reproducibility lowers failure risk (especially in safety-critical domains like healthcare, autonomous vehicles, finance), which can translate to reduced expected damages and may lower insurance premiums or compliance costs.
  • Better reproducibility supports auditability and regulatory compliance (e.g., documenting bug replication steps), increasing the economic value of AI offerings in regulated sectors.
• Market and industry effects:
  • Increased demand for specialized debugging/reproduction platforms and datasets (benchmarks) — companies providing such tooling or managed services could capture value.
  • Potential vendor lock-in: reliance on specific LLMs/agent stacks may centralize market power to major LLM/service providers, affecting bargaining and pricing.
  • Substitution/augmentation in labor markets: some routine reproduction tasks can be automated, altering the nature of entry-level ML engineering roles; but higher-value roles (model design, causal analysis) may grow.
• Investment and R&D allocation:
  • Firms may reallocate R&D and QA budgets toward model feature development when reproduction overhead falls, accelerating product feature cycles and shifting investment incentives.
• Caveats / limits impacting economic estimates:
  • External validity: evaluation on 106 bugs from 16 projects is promising but not exhaustive; large-scale, diverse production systems may present additional costs/complexities.
  • Nondeterminism and environment coupling remain fundamental issues: full reproduction sometimes requires precise hardware/frame versions or proprietary data; RepGen reduces but does not eliminate these frictions.
  • Ongoing costs: compute and LLM API usage are recurring; cost-benefit depends on scale of debugging workload and LLM pricing.
  • Risk of over-reliance: automated agents might reproduce bugs but not always diagnose root causes or ensure safe fixes; supervision and human-in-the-loop remain necessary.
• Net economic takeaway:
  • RepGen-style tools can materially lower the marginal cost of reproducing and triaging DL bugs, reduce downstream failure costs, and speed product cycles — all of which increase the effective productivity and reduce operational risk in AI development. However, organizations must factor in provider, compute, and integration costs, and guard against vendor concentration and over-automation risks.

If you want, I can (a) extract the key numeric results into a one-page table for quick reporting, (b) outline a simple back-of-the-envelope model estimating cost savings for an ML team using RepGen (given per-hour developer rates and bug volumes), or (c) summarize the technical pipeline as a diagram-friendly bullet list for presentations. Which would help most?