Small, agentic language models—boosted by multi-sampling and compiler feedback—can match or beat larger models at optimizing telescope software, cutting the compute and energy needed to port code to accelerators; this offers a practical, lower-footprint route to meeting LOFAR’s 40× computational challenge.

Main Finding

Multi-sampling small language models (SLMs) combined with an agentic, tool-feedback loop (including Retrieval-Augmented Generation and compiler output) can match or exceed single-generation large LLMs on repository-level code-completion tasks while substantially reducing compute resources and the associated environmental cost. Feeding compiler errors back into the model consistently improves correctness by ~3–4 percentage points across tested models.

Key Points

• Problem context: LOFAR radio-astronomy software faces a ~40× increase in computational needs from upcoming upgrades; software optimization and porting to accelerators (GPUs, etc.) are critical to meet energy budgets.
• Two complementary approaches to sustainable AI-assisted code optimization:
• Multi-sampling SLMs: run many generations from a small model within the same generation-time budget as a single run of a larger model, then select the best candidate.
• Agentic feedback loop: give models access to domain documents via RAG and to tool outputs (compiler/interpreter, unit tests, static/dynamic analysis) in a revision loop so models iterate on errors.
• Empirical claims:
  • Multi-sampling Qwen2.5-Coder-7B (k samples) can match or slightly outperform Qwen2.5-Coder-32B while using a fraction of memory/GPU resources (paper cites a 4× resource reduction example).
  • Incorporating compiler feedback into the loop yields consistent improvements of ~3–4 percentage points in performance across models.
• Selection and reranking: naive selection by highest log-prob (logP) is used for candidate choice; an upper-bound oracle (selecting candidate by highest cosine similarity to ground-truth) shows room for improved reranking (e.g., via unit tests or execution-based verification).
• Human-in-the-loop design: SuperRAG is intended to produce targeted suggestions (identify bottlenecks and advise fixes) rather than automatically committing code, to reduce security risks and technical debt.

Data & Methods

• Primary benchmark: CrossCodeEval (multilingual, repository-level cross-file code completion) in the rg1 setting with cross-file context retrieved using UniXCoder similarity.
• Experimental design for multi-sampling vs single-generation:
  • Paired comparisons using 10 random seeds → 10 groups of 150 tasks per language (Python and C#).
  • 32B model: one completion per task within a generation-time budget.
  • 7B model: k completions per task calibrated so total generation wall-clock time did not exceed the 32B model’s generation time (k = 7 for Python, k = 6 for C#).
  • Hardware used in reported experiments: 7B on one A100-SXM4-40GB GPU; 32B on four A100-SXM4-40GB GPUs (memory constraints required multi-GPU for 32B).
  • Evaluation metric: character n-gram cosine similarity between generated completion and ground truth; also computed best-cosine upper bound (oracle selection).
• Agentic feedback loop:
  • Core idea: integrate RAG (documents, codebase context) and tools (compiler/interpreter, unit tests, static/dynamic analysis) in an iterative revision loop; feed tool outputs (e.g., compile errors) back to model as context to guide fixes.
  • Prototype experiment used an AMD EPYC 9255 system (48 cores, 768 GiB RAM) without GPUs.
  • Base models tested in feedback experiments: Qwen2.5-Coder-3B-Instruct, Qwen3-4B, Qwen3-Coder-Next (MoE 80B-A3B), CodeLlama-7b-Instruct-hf, CodeGemma-7b-it.
  • Observed improvement from compiler feedback: consistent +3–4 percentage points in tested metrics.
• Reproducibility: code released at https://github.com/SuperCode-Leiden-University/superRAG/

Implications for AI Economics

• Lowering operating/energy costs: Multi-sampling SLMs that match larger-model performance under the same time budget can substantially reduce GPU/memory requirements and inference energy, directly lowering operational expenses (electricity, GPU hours) for large-scale scientific code optimization and for ongoing maintenance of scientific pipelines.
• Better cost–performance trade-offs: The study shows a practical path to improve cost-effectiveness of AI-assisted engineering: invest in orchestration (multi-sampling, retrieval, tool feedback, reranking) rather than relying solely on bigger models. This shifts capital allocation from larger model licenses / bigger GPU clusters to engineering of tooling and verification.
• Reduced externalities and compliance with sustainability targets: For projects (like LOFAR/LENSS) with fixed energy budgets, SLM + agentic workflows can help meet ambitious scaling targets (e.g., 40× compute demand) without proportional increases in energy consumption—supporting public good science while aligning with climate/energy constraints.
• Labor and productivity effects: AI-assisted suggestion workflows that avoid auto-committing code and keep humans in the loop may increase programmer productivity while keeping control and reducing security/technical-debt risk—altering the marginal productivity of software engineers in scientific domains.
• Market & procurement implications: Organizations may prefer smaller-model deployments and invest in toolchains (RAG indices, test harnesses, static/dynamic analysis infrastructure) rather than premium large-model APIs; procurement strategies should account for total-cost-of-ownership including inference energy and toolchain development.
• Policy and incentive design: Grants, procurement rules, and sustainability assessments for big-science projects should measure lifecycle AI inference costs and incentivize approaches (like SLM multi-sampling and agentic tooling) that lower operational emissions and costs.
• Limitations and risks with economic relevance:
  • Overhead of engineering the agentic pipeline (building RAG indices, test suites, and tool integrations) has upfront costs; cost-benefit depends on codebase size and reuse.
  • Current experiments use naive selection (logP); achieving oracle-level gains requires investment in rerankers, test generation, or execution-based verification—again implying upfront engineering expenditure.
  • Model inference still carries nontrivial energy costs—SLMs reduce but do not eliminate this; widespread adoption requires careful accounting of these remaining externalities.

Overall, the paper indicates a promising, economically attractive direction: focusing investments on smarter orchestration (multi-sampling, RAG, tool feedback, reranking) and human-in-the-loop workflows can yield good code-quality/performance outcomes with substantially lower compute and energy costs than naively scaling model size.