Edge-sized LLMs can diagnose 5G faults reliably and cheaply, but they stumble on precise standards recall; multi-judge LLM scoring proves consistent enough to automate free-text telecom evaluation, though human validation remains absent.

Main Finding

Lightweight, edge-deployable LLMs can reliably perform free-text 5G fault diagnosis (>=90% on the fault benchmark) while remaining far cheaper and lower-latency than heavier frontier models. However, precise zero-shot recall of 3GPP / O-RAN specifications remains a major gap (all models <60% on the specification benchmark). Using three independent frontier models as automated judges yields high inter-judge agreement (mean ≥ 0.90), supporting the LLM-as-Judge approach for scalable free-text evaluation in telecom.

Key Points

• Models evaluated (student models): Claude-Haiku-4.5 (Anthropic), GPT-5.4-Mini (OpenAI), Gemini-3.1-Flash-Lite (Google).
• Judge models (LLM-as-Judge): GPT-5.5, Gemini-3.1-Pro-Preview, Gemini-3.5-Flash.
• Benchmarks (100 samples each, converted to free-text):
  • TeleQNA ORAN FT (3GPP / O-RAN specification knowledge) — hardest.
  • 5G-Faults FT (realistic fault scenarios) — easiest for students.
  • TeleInter FT (technical/interview-style questions).
• Accuracy (mean across judges, per student model):
  • GPT-5.4-Mini: 82.1% overall (best).
  • Gemini-3.1-Flash-Lite: 78.1% overall.
  • Claude-Haiku-4.5: 71.0% overall.
• Benchmark-specific highlights:
  • TeleQNA ORAN FT: all models <60% (weak zero-shot spec recall).
  • 5G-Faults FT: all models ≥90%; GPT-5.4-Mini reached 98.3%.
  • TeleInter FT: GPT-5.4-Mini & Gemini ~90+, Claude ~76%.
• LLM-as-Judge reliability:
  • Pairwise inter-judge agreement rates ranged ~0.86–0.99; mean agreement ≥0.90 across runs.
  • Notable divergence: GPT-5.5 graded Claude-Haiku-4.5 lower on 5G-Faults FT (pairwise agreement 0.86 vs Gemini judges), showing provider-level grading style differences.
• Operational metrics (approx. per 100-sample run):
  • API cost: Gemini ~ $0.09; GPT-5.4-Mini ~$0.16–0.34; Claude ~$0.42–0.44.
  • Latency (mean per sample): Gemini 3.4–4.3s; GPT up to ~9.1s; Claude up to ~12.3s.
  • Token behavior: Claude emits most reasoning tokens; Gemini emits the fewest completion tokens (more concise).

Data & Methods

• Pipeline:
  • Inspect AI-based evaluation with a solver stage (prompt templating, chain-of-thought enforcement, generate) and a scorer stage invoking three judges in parallel.
  • All calls routed via OpenRouter for unified accounting.
• Prompting:
  • Structured system prompt: require two-part output:
    • Student Explanation: (2 lines)
    • Student Answer: (1 line)
  • Zero-shot generation (no few-shot, RAG, or tools).
• Free-text evaluation:
  • Reference answer provided to judges as a semantic guide only.
  • Judges return binary grades: Correct (C=1) or Incorrect (I=0), plus a one-sentence justification.
• Metrics collected:
  • Per-judge accuracy, mean accuracy across judges.
  • Pairwise inter-judge agreement (probability two judges assign same binary grade).
  • Token counts (reasoning vs completion), latency per sample, and total API cost per run.
• Data:
  • Three telecom-specific datasets (TeleQNA_ORAN_FT, 5G-Faults_FT, TeleInter_FT), 100 Qs each, open-sourced on Hugging Face (rishieee namespace).
• Experiments:
  • Nine runs (3 student models × 3 datasets); deterministic sampling, full logs persisted.

Implications for AI Economics

• Cost vs. accuracy trade-offs favor lightweight models for operational deployment:
  • Gemini-3.1-Flash-Lite shows the best accuracy-to-cost and latency profile — attractive for distributed, edge-level inference across many network elements.
  • Very low per-run inference costs (order of $0.09 per 100 samples in this setup) indicate that large-scale automated diagnostics can be economically feasible.
• Energy and latency savings matter for telco operators:
  • Lower-latency, lower-token models reduce energy use and enable real-time/near-real-time diagnostic workflows at the edge, lowering operational expenditures (OPEX).
• Task suitability and risk management:
  • High performance on fault analysis suggests automation can reduce human operator load and mean time to repair, implying operational cost savings.
  • Poor zero-shot spec recall (TeleQNA ORAN FT) signals risk: specification-critical tasks likely require retrieval augmentation (RAG), domain fine-tuning, or human-in-loop verification to avoid costly errors—these add development/maintenance costs.
• Evaluation economics:
  • Multi-judge LLM-as-Judge can dramatically cut labeling/evaluation costs compared to human grading while delivering high inter-judge agreement, supporting cheaper iterative benchmarking and model selection.
  • Provider-level grader differences (observed systematic divergence) create a potential source of evaluation risk; buyers should use multiple judges and audit samples to avoid procurement based on biased single-judge scores.
• Vendor and deployment considerations:
  • Choice of student model/provider affects total cost of ownership (inference pricing, latency SLAs, token behavior). Operators must weigh marginal accuracy gains versus per-query cost and latency.
  • Lightweight models lower barriers to distributed deployment (privacy, bandwidth, latency), influencing where intelligence is placed (edge vs. centralized).
• Strategic recommendations:
  • Use lightweight LLMs for first-line automated fault triage and suggested remediations, with escalation to humans or stronger models for spec-sensitive actions.
  • Invest in retrieval-augmented or domain-fine-tuned variants to close the specification-recall gap before fully automating compliance-critical decisions.
  • Employ multi-judge LLM evaluation as a cost-effective, reproducible evaluation pipeline, but periodically validate against human experts to detect grader drift or provider biases.