AI agents can generate large numbers of high-quality multiple-choice questions, but they do not yet match expert-authored items on depth or difficulty calibration; researchers' roles shift away from item writing toward specifying, supervising and auditing AI-driven pipelines.

Main Finding

An AI-orchestrated pipeline in which a single researcher coordinated multiple LLM agents can generate large sets of SAT-style math MCQs that score very highly on automated, LLM-based quality rubrics (means ≈ 4.64–4.89/5). However, generated items are not fully equivalent to human, expert-vetted baseline items on many pedagogically important criteria: strict equivalence (24/24 criteria) never occurred, and only 9–12 of 24 criteria met practical equivalence depending on judge and analysis track. The researcher’s labor shifted away from writing items toward specification, orchestration, verification, and governance — suggesting emerging “AI research operations” tasks and new labor demands in AI-augmented scientific workflows.

Key Points

• Scale and materials
  • Baseline: ~1,071 SAT Math MCQs collected from College Board PDFs; filtering produced final corpora (e.g., 1,013 items per one extractor).
  • Open-source grounding: OpenStax textbooks converted and chunked into 2,980 pedagogically coherent chunks.
• Models and roles
  • Two LLMs used as generators: GPT-5-nano and Gemini-2.5-Flash (Feb 2026 access).
  • The same two LLMs used as independent judges (dual-judge design) to score items on a 24-criterion rubric.
• Retrieval & grounding
  • Each baseline MCQ mapped to textbook chunks via embeddings. Combined relevance score = 0.3×metadata_sim + 0.7×content_sim.
  • Retrieved chunks used as grounding prompts for conditional MCQ generation.
• Evaluation rubric & method
  • 24 detailed criteria across 6 thematic categories (Accuracy & Grounding; Clarity & Presentation; Distractors & Answer Quality; Difficulty & Engagement; Domain & Skill Alignment; Difficulty & Cognitive Calibration).
  • Two analysis tracks: matched (paired on same question_id) and whole-set.
  • Equivalence testing via two one-sided tests (TOST); definitions: strict similarity = 24/24 criteria equivalent; practical similarity ≥19/24 criteria.
• Quantitative outcomes
  • High average ratings across conditions (examples): SAT baseline 4.80 (Gemini judge) / 4.64 (OpenAI judge); Gemini-generated 4.89 / 4.68; GPT-generated 4.82 / 4.66.
  • Large share of criterion scores at the maximum (74.1%–93.4% of individual criterion ratings = 5 depending on judge).
  • Persistent, consistent gaps concentrated on deeper pedagogical dimensions: skill depth, cognitive engagement, difficulty calibration, and metadata alignment.
  • Equivalence: strict similarity never achieved; only 9–12/24 criteria reached equivalence depending on judge and track.
• Work-practice shift
  • Primary human activity became specification (designing constraints and rubrics), orchestration of agents, building validation loops, diagnosing recoveries for tool failures, and auditing provenance.

Data & Methods

• Data sources
  • Baseline MCQs: College Board SAT Math question bank PDFs (initially 1,682 items; 1,071 identified as MCQs; filtered to ~1,013–1,027 depending on extractor).
  • Grounding corpus: OpenStax open textbooks (downloaded as PDFs).
• Corpus processing
  • PDF → structured formats (JSON/Markdown) via LLM-augmented conversion tools.
  • Chunking: 2,980 section-based chunks preserving pedagogical coherence (to avoid orphaned equations/refs).
• Mapping & retrieval
  • Each baseline MCQ represented by metadata query (domain/skill) and content query (stem and rationale).
  • Embedding-based retrieval with weighted similarity: 0.3 metadata + 0.7 content.
• Generation
  • Prompted LLMs (GPT-5-nano, Gemini-2.5-Flash) received retrieved grounding text plus metadata (domain, skill, difficulty, Bloom cognitive level) to produce new MCQs.
  • Produced item sets: Gemini-generated (~1,065 items), GPT-generated (~1,071 items) — in addition to the baseline set.
• Evaluation
  • 24-criterion rubric (1–5 Likert with detailed descriptors); criteria grouped into 6 themes.
  • Two independent LLM judges (the generator models reused as evaluators) scored all three sets.
  • Statistical analysis: descriptive summaries; criterion-level diagnostics; equivalence testing using TOST in matched and whole-set tracks.
• Limitations noted by authors
  • Judges are LLMs (not human raters); potential evaluator-model alignment biases (generators may match judges’ stylistic preferences).
  • No student-administration data (no direct validation against learner responses or classical psychometric measures).
  • Single-researcher pilot; domain limited to SAT Math and to open-textbook grounding.

Implications for AI Economics

• Task re-allocation and labor demand
  • Substantial substitution of routine item-authoring tasks by LLMs; simultaneous creation of new, higher-level tasks: specification engineering, pipeline orchestration, verification/audit, provenance management, and prompt/rubric design.
  • Demand shifts toward workers skilled in “AI research operations” (tooling, evaluation design, error recovery), increasing returns to those coordination/oversight skills while reducing demand for pure item-authoring labor.
  • Potential for occupational polarization: fewer routine producers, more specialized managers/validators.
• Productivity and cost effects in education and test-prep markets
  • Large-scale, low-cost MCQ generation could lower content production costs and enable rapid expansion of practice materials; this may compress margins for incumbents who monetize content creation.
  • If quality gaps on deep cognitive calibration persist, high-stakes providers may still value human experts for final vetting; but mid- and low-stakes markets could see swift automation-driven substitution.
  • Market concentration risk: major model providers (or integrators) could capture downstream content-production capabilities, influencing pricing and distribution (e.g., free practice tests bundled with major LLM offerings).
• Quality, validity, and externalities
  • High surface-level quality (grammar, clarity, plausible distractors) may mask substantive gaps in assessment validity (depth, difficulty calibration). Economically, this suggests a risk of eroding signal quality of assessments if deployed without adequate human-in-the-loop validation.
  • Reliance on LLM-based judges may overestimate readiness — evaluator-model alignment can bias quality assessments upward, producing false confidence and negative downstream educational or reputational externalities.
• Governance, regulation, and certification
  • Need for provenance, audit trails, and standardized validation protocols to certify AI-generated assessment materials. These requirements create new compliance and quality-assurance services (a potential niche market).
  • Regulators and accreditation bodies may demand human expert validation for high-stakes instruments, slowing full automation adoption in regulated contexts.
• Industry structure and competition
  • Lower content costs enable new entrants (edtech startups, platforms) to scale offerings rapidly; incumbents must adapt by investing in human-AI workflows and verification capabilities.
  • Vertical integration (models + content distribution) creates strategic advantages for large AI providers; policy concerns include competition, quality oversight, and fair access.
• Policy and workforce implications
  • Policymakers should anticipate transitional frictions: worker displacement in content-creation roles, need for reskilling into oversight/ops roles, and unequal geographic impacts if orchestration skills concentrate in tech hubs.
  • Public investment in standards, transparent evaluation practices, and training for verification/audit competencies can mitigate social costs and improve diffusion of safe, validated AI-generated educational materials.

Summary takeaway: LLM orchestration can produce large volumes of high-looking educational items and materially change the nature of research/content-production labor, but substantive pedagogical gaps remain. Economically, the technology drives productivity and reshapes skill demand (toward orchestration and governance), while raising quality-control, market-structure, and regulatory challenges that will determine how benefits and risks are distributed.