Autonomous LLM research agents find near–state-of-the-art telecom retrieval models quickly and cheaply—reaching 0.343 Recall@1 (≈90% of internal SoTA) in 10 weeks and $150–$200 of API cost—yet lack the creativity of human teams and still need human direction.

Main Finding

Autonomous research agents can rapidly produce competitive solutions for an open-ended, industry-grade ML task (telecom ticket retrieval). A mostly automated campaign produced a single fine-tuned retriever that achieved 0.343 Recall@1 — about 90% of an internal human-engineered SoTA (0.380) — in far less calendar time (10 weeks vs. ~10 months of human development) and at modest API cost (~$150–$200 per Cursor campaign). However, agents are best at deep, narrow hyperparameter/data-sweep optimization, lack human-like intuition/creativity (e.g., designing novel ensembling, re-ranking, or data-augmentation strategies), and require nontrivial operational “babysitting” and harnessing.

Key Points

• Performance
  • Best discovered model: single fine-tuned MPNet retriever.
  • Metrics (selected): discovered model Recall@1 = 0.343, Recall@10 = 0.630, Recall@50 = 0.768.
  • Comparison: internal human SoTA Recall@1 = 0.380; baseline MPNet (human fine-tuned) Recall@1 = 0.200; BM25 = 0.072.
• Cost & Time
  • Typical Cursor Composer 2.5 campaign (10–20 experiments): $150–$200 in API charges.
  • Claude Sonnet 5 campaigns: ~2–3× higher ($300–$600).
  • Open-weight model (GPT-OSS) can be hosted locally (no API fees) but operational reliability varies.
  • Compute: ~1 GPU-week for the 17-experiment campaign; elapsed calendar time ~10 weeks end-to-end.
  • Human-engineered systems took ~10 months.
• Agent & Framework Effects
  • Little difference in peak downstream performance across: single-agent vs multi-agent frameworks; different LLM backends (Cursor, Claude, GPT-OSS); and whether agents were informed about human SoTA components.
  • Agents consistently excelled at parameter sweeps but rarely invented the creative, structural additions used in human SoTA (e.g., ensembling, re-ranking, generative augmentation).
• Operational Practicalities
  • Successful deployment required a harness: explicit task/environment docs, explicit search space definitions, and a scripted outer loop (to avoid stalled agents).
  • Agents underutilized compute and sometimes failed to orchestrate sub-agents reliably (open-weight models particularly brittle).
  • Autonomous runs need human oversight for stability, debugging, and interpreting failures.

Data & Methods

• Task: Telecom Ticket Retrieval (retrieve correct resolution documents given incident tickets; evaluation = cluster-based Recall@K).
• Dataset: ~250k incident reports (tickets: T), 204k fault analyses (FAs), 89k technical analyses (TAs). Evaluation set: 7.6k held-out queries, 1.2k held-out gold resolution documents; retrieval against full corpus.
• Models & Baselines: BM25; fine-tuned MPNet (human baseline); ensemble (Trabelsi & Uzunalioglu 2026); internal SoTA (human system with data augmentation, ensembling, re-ranking).
• Autonomous research implementations:
  • Two framework styles: single-agent loop (Karpathy-style autoresearch) and multi-agent team (AutoScientists-style).
  • LLM backends: Cursor Composer 2.5, Claude Sonnet 5, GPT-OSS 120B (locally hosted).
  • Informedness: uninformed (start-from-scratch) vs informed (documentation enumerated human SoTA components).
  • Harnessing: explicit documentation of task/resources, explicit search-space variables (representation, data generation, architecture/hyperparams), and a deterministic bash outer loop to resume agent sessions.
• Hardware: training on 4× NVIDIA RTX A6000 servers (48 GB VRAM each).
• Experiment scale: best campaign completed 17 unique experiments; reported costs and runtimes as above.

Implications for AI Economics

• Labor substitution and productivity
  • Short-run: Autonomous agents can substitute for routine, time-consuming parts of applied ML R&D (hyperparameter sweeps, systematic representation choices), accelerating product development cycles and reducing calendar-time costs.
  • Role-shift rather than full replacement: Human researchers remain essential for creative, structural innovations (ensembling, re-ranking, augmentation design), for handling operational failures, and for high-level direction. Economically, value shifts from manual experiment execution to higher-level oversight, curation, and strategy.
• Cost structure and firm choices
  • Marginal cost of an autonomous campaign is modest (hundreds of dollars in API fees or primarily capital for hosting open-weight models). This makes automated exploration affordable even for mid-sized firms, lowering barriers to iterate.
  • Fixed costs: building a robust harness, instrumenting experiments, and human oversight represent nontrivial fixed setup costs; these amortize over repeated campaigns.
  • Hosting trade-offs: open-weight models reduce API fees (lower variable cost) but can increase engineering/operational risk and require infra capital.
• Returns to scale and R&D allocation
  • High returns where the search problem is large but structurally local (many hyperparameters or data-op variants): automated agents can rapidly produce near-SoTA gains at low marginal cost.
  • Diminishing returns for open-ended creativity tasks: economic value of human expertise remains high where novelty and cross-domain intuition are required.
  • Firms can optimize R&D allocation by using agents for intensive local search (cheap, fast) and allocating senior researchers to architectural strategy, novel methods, and assimilation of agent outputs.
• Market and workforce effects
  • Demand shift: increased demand for skills in agent orchestration, experiment harnessing, and ML systems engineering; decreased demand for purely experimental grind work.
  • Democratization and competition: accessible automated campaigns (via cheap API or open-weight hosting) can lower entry costs for applied ML, increasing competition in domains where systematic tuning matters.
• Risk, governance, and reproducibility
  • Operational brittleness and nondeterministic agent behavior introduce governance and reproducibility challenges; firms need protocols for verification and human review to avoid costly downstream errors.
  • Economic risk arises if firms over-rely on agents for open-ended innovation; investments should maintain human expertise to mitigate long-term stagnation in breakthrough methods.

Actionable recommendations for firms/policymakers - Integrate autonomous agents into a hybrid human-in-the-loop workflow: use agents for exhaustive, repeatable sweeps; reserve humans for creative design and oversight. - Invest in reproducible harnessing and tooling as a fixed-capital expenditure to amortize across many campaigns. - Consider open-weight model hosting to lower variable costs, but budget for increased engineering and operational monitoring. - Re-skill researchers toward agent orchestration, experiment design, and interpretability to capture productivity gains while retaining innovation capacity.