Adaptive prompt selection slashes LLM grading costs by roughly three-quarters without losing accuracy: a bandit-driven grader matched grid-search agreement with human IELTS scores while dramatically cutting calls and tokens.

Main Finding

Treating prompt–model combinations as arms in a multi-armed bandit (MAB) controller lets an AES system discover and concentrate on cost-effective prompting strategies online. On IELTS Task 2 essays the bandit approach matched exhaustive grid-search accuracy while cutting LLM calls by ~78% and token use by ~73%, with the best-performing recipe being a multi-step, example-calibrated prompt.

Key Points

• Problem: prior LLM-based AES work uses static prompt selection (grid search) that ignores operational costs and changing optima.
• Proposal: cast each prompt–model recipe as an arm in an epsilon-greedy MAB; reward is shaped by scoring accuracy (negative absolute error) and can include token-cost penalties.
• Recipes tested (4): multi-step with examples, multi-step without examples, single-step with examples, single-step without examples. Multi-step decomposes IELTS criteria (Task Response; Coherence & Cohesion; Lexical Resource; Grammatical Range & Accuracy) then computes an overall score.
• Best recipe: multi-step with calibration examples. MAB allocated ≈70% of pulls to it after exploration stabilized (~step 100).
• Surprising ablation: simplified prompts (no long rubric descriptions) improved performance and reduced tokens versus detailed rubric prompts — e.g., Multi-Step+Examples (simplified) MAE 0.862, QWK 0.566 vs detailed-rubric MAE 0.965, QWK 0.485.
• Accuracy summary (approximate from experiments):
  • Multi-step + examples: MAE ≈ 0.85–0.86, QWK ≈ 0.55–0.57
  • Single-step + examples: MAE ≈ 1.0
  • Single-step no examples: MAE ≈ 1.45
  • Multi-step no examples: MAE ≈ 1.55 (worst)
• Cost/resource savings (Table 1):
  • LLM calls: Grid search = 7,870; MAB = 1,697 → 78.4% fewer calls
  • Tokens: Grid search ≈ 10.9M; MAB ≈ 3.0M → 72.8% fewer tokens
  • Estimated experiment cost: Grid ≈ $1.4 vs MAB ≈ $0.4 (~70% cost reduction)
• Other metrics tracked: latency, per-call token usage, and learning curves (first cost–reliability curves in AES context).
• Implementation: Google Gemini Flash 2.5 used in experiments; code and policy logs planned for release.

Data & Methods

• Dataset: IELTS Writing Scored Essays (Kaggle), Academic Task 2 subset — 787 essays with band scores 1–9.
• Prompting:
  • Dynamic prompt assembly module (prompt library + runtime injection of essay and examples).
  • Four recipes combining single-step vs multi-step and with vs without calibration examples.
• Bandit controller:
  • Epsilon-greedy MAB (ϵ = 0.2 fixed) balancing exploration/exploitation.
  • Reward = negative absolute error (± optional token cost penalty).
  • 500-step experiment reported; arms pulled according to learned policy.
• Baseline: exhaustive grid search over recipes (apply all recipes to all essays).
• Model: Google Gemini 2.5 (chosen for performance/latency/cost).
• Evaluation metrics: Mean Absolute Error (MAE) against human scores, Quadratic Weighted Kappa (QWK), token consumption, latency, and estimated API cost.
• Ablation: compared prompts with detailed rubric descriptions vs simplified prompts (no rubric text).

Implications for AI Economics

• Operational cost reduction: Adaptive selection can sharply lower token and API-call budgets for large-scale AES deployments, directly reducing operating expenses for edtech and testing organizations.
• Dynamic pricing & model updates: Framing prompt choice as an online learning task lets systems adapt to model upgrades, price changes, or shifts in essay populations — reducing the need for repeated offline re-tuning.
• Cost–reliability tradeoffs: Platforms can formally trade accuracy for savings by incorporating token/call penalties into reward signals or by dynamically switching to cheaper recipes for low-stakes use cases.
• Product design recommendations:
  • Instrument per-call tokens, latency, and reliability metrics (MAE/QWK) and use them jointly in selection policies.
  • Implement bandit or contextual-bandit routing (contextual variants can exploit essay features to further improve efficiency).
  • Use human review or conservative routing for borderline/high-stakes items; bandits are good for average-case efficiency but need oversight for fairness/robustness.
• Market & supply effects: reducing average per-essay compute demand can lower marginal costs and enable scaling (more customers, lower prices) — but widespread adoption could shift LLM provider revenue patterns and unit pricing strategies.
• Risks and externalities:
  • Bandits can entrench subtle biases if reward signals reflect biased human labels or if cheaper prompts exhibit systematic errors across subgroups — must monitor fairness metrics and ensure psychometric validity.
  • Adaptive controllers require auditability and logging (policy logs, pulls per arm) for regulatory compliance in high-stakes assessment.
• Research & investment priorities from an economics perspective:
  • Evaluate contextual bandits and hybrid strategies (cheap routing + expensive fallback) for marginal gains.
  • Study long-run effects of adaptive routing on pricing negotiations with LLM vendors and on total compute demand.
  • Quantify welfare gains from decreased testing costs (expanded access, new market segments) versus risks from potential degradation of measurement quality for minority groups.

Short actionable takeaways for practitioners: - Start by instrumenting token and accuracy metrics; run a small-scale epsilon-greedy bandit to identify promising recipes. - Include token/cost penalties in the reward if cost savings are a priority. - Use multi-step, example-calibrated prompts as a strong default but test simplified prompts too — more rubric text is not always better. - Add human-in-the-loop checks and fairness monitoring before deploying in high-stakes contexts. - Explore contextual bandits next (essay features, candidate metadata) to capture per-item heterogeneity and improve efficiency further.