Smaller LLMs can outcompete flagship models for software-test review: GPT-4o Mini achieved the best accuracy (6.07 MAAE), high first-attempt reliability (96.6% ECR@1) and a low operating cost ($1.01 per 1K), yielding roughly a 78x cost reduction versus a high-reasoning GPT-5 configuration; overall costs ranged from $0.45 to $78.96 per 1K depending on model and reasoning strategy.

Main Finding

LLM-as-a-Judge (LAJ) can produce accurate, structured, rubric-aligned assessments of Gherkin acceptance-test coverage at production scales, but model choice and operational reliability materially affect deployment cost and accuracy. In the authors’ benchmark, a smaller model (GPT-4o Mini) delivered the best overall balance: highest accuracy (MAAE = 6.07 pp), high first-attempt reliability (ECR@1 = 96.6%), and low adjusted cost ($1.01 per 1K evaluations), giving ~78× cost advantage versus a high-reasoning GPT-5 configuration while slightly improving accuracy.

Key Points

• LAJ framework: rubric-driven prompt + strict JSON output that returns coverage percentage, coverage analysis, and rubric-aligned flags/recommendations.
• Benchmark: 100 expert-annotated Jira stories + generated Gherkin scripts (Kill Bill domain); ground-truth from 3 senior QA annotators using a 4-dimension weighted rubric (scenario completeness 40%, acceptance alignment 30%, HTTP-specific 20%, assertion quality 10%).
• Evaluation protocol: 20 model configurations (GPT-4 family, GPT-5 family with low/med/high reasoning settings, and two open-weight families at various reasoning levels) × 100 cases × 5 runs = 500 evaluations; tracked accuracy, reliability, token usage, retries, and costs.
• New operational metric: Evaluation Completion Rate (ECR@1) — percent of evaluations that succeed (valid, parseable JSON) on first attempt. They also define an adjusted cost per 1K that scales nominal cost by 100/ECR@1 to account for retry overhead.
• Main quantitative results:
  • Best model: GPT-4o Mini — MAAE = 6.07±0.08, APS ≈ 93.93%, PMR 32.6%, ECR@1 = 96.6±2.2%, adjusted cost = $1.01/1K.
  • Cost span across models: $0.45 to $78.96 per 1K (≈175× range).
  • ECR@1 span: 85.4% (worst open-weight config) to 100% (several proprietary configs).
  • GPT-5: higher reasoning generally improves accuracy but increases cost; reliability can be high at low/medium reasoning as well.
  • Open-weight models: lower nominal costs but substantially worse accuracy (MAAE 14–19 pp for many configs) and often lower reliability; increasing reasoning usually degraded open models.
• Failure modes: malformed JSON, missing fields, occasional API timeouts — these drive retries and increase adjusted cost.
• Reproducibility: dataset, framework, and code released (GitHub link in paper).

Data & Methods

• Dataset construction:
  • 100 Jira tickets authored by experienced product owners, distributed across HTTP methods (GET 50%, POST 21%, DELETE 15%, PUT 14%).
  • For each ticket: Gherkin acceptance tests were generated via an automation pipeline leveraging GPT-4 (used for script generation).
  • Ground truth: 3 senior QA engineers annotated each script using a weighted 4-dim rubric, scoring 0–10 then normalized to 0–100.
• Models evaluated:
  • Proprietary: GPT-4 variants (4o, 4o Mini, 4.1, 4.1 Mini, 4.1 Nano) and GPT-5 family (base, mini, nano; each with low/med/high reasoning).
  • Open-weight: GPT-OSS 20B and 120B, each at low/med/high reasoning.
• Metrics:
  • Accuracy: Mean Absolute Assessment Error (MAAE), Assessment Performance Score (APS = 100 − MAAE), Perfect Match Rate (PMR), Close Match Rate (CMR ≤ ±5 pp).
  • Reliability: ECR@1, mean attempts per valid evaluation.
  • Cost: token-based cost per evaluation aggregated to cost per 1K (C1K); adjusted cost C1K,adj = C1K × (100 / ECR@1).
• Protocol: identical prompts across models, 5 independent runs to compute mean ± std for each metric.

Implications for AI Economics

• Procurement and pricing decisions must include reliability, not just per-call pricing or accuracy. ECR@1 directly converts reliability failures into added operational cost via retries; a lower nominal per-call price can be offset by higher retry overheads.
• Marginal returns to “reasoning effort” vary by family and size:
  • For high-capacity proprietary models (GPT-5), increasing reasoning yields predictable accuracy gains but with large cost increases — firms should explicitly trade off marginal accuracy vs marginal cost (e.g., moving GPT-5 from high→low reasoning saved ∼70% cost for ∼1.5 pp accuracy loss).
  • For open-weight models, added reasoning produced no consistent accuracy gain and often degraded reliability — tuning reasoning parameters is essential and can be counterproductive.
• Smaller specialized/configured models can dominate on cost-effectiveness. The best production choice in this study (GPT-4o Mini) is both cheaper and slightly more accurate than a much more expensive GPT-5 high-reasoning option. This suggests vendors and enterprises should benchmark lightweight variants for judge-style evaluation tasks rather than assuming largest models are best.
• Budgeting/forecasting for CI/CD-scale workloads should use adjusted cost formulas that incorporate expected ECR@1 and mean attempts, and should run sensitivity analyses across reasoning settings. Example from paper: for 100k evaluations/month, costs ranged from ~$45 (cheap open-weight) to ~$7,896 (GPT-5 high); GPT-4o Mini costs ~$101/month in that scenario.
• Market implications: demand for “judge”-optimized models with strong instruction-following and structured-output reliability (and low-cost variants) will likely grow. Vendors may compete on reliability/SLA as much as raw throughput or token price.
• Risk management & SLAs: organizations deploying LAJ-like systems should include reliability/format-following SLAs (ECR@1 targets), monitoring for malformed outputs, and fallback/retry strategies that are cost-aware.
• Research & evaluation investments: releasing a reproducible dataset and framework enables standardized economic benchmarking (cost × reliability × accuracy), which should become common practice before large-scale procurement of LLM services.

Limitations to bear in mind for economic decisions - Benchmark is domain-specific (Kill Bill-style billing APIs); performance and optimal choices may differ for other application domains or test styles. - Gherkin scripts were generated with an LLM in the pipeline, which could bias the evaluation distribution versus human-authored tests. - Ground truth from three annotators is high-quality but limited in scale; larger/more diverse annotator pools may shift the effective MAAE baseline.

Overall, the study shows the importance of combining accuracy, structured-output reliability (ECR@1), and adjusted cost when evaluating LLM deployment choices for automated judgment tasks in software QA.