Short, researcher-led edtech pilots overstate classroom benefits: learning gains depend on teachers' orchestration and clear mathematical purpose, not gadget sophistication. Generative AI magnifies unresolved questions about who validates mathematical reasoning, so procurement should fund teacher training and rigorous, classroom-embedded evaluations rather than tool purchases alone.

Main Finding

Current evidence on digital technology in school mathematics is fragmented and often biased toward short, small-scale, researcher-led interventions; meaningful learning gains depend less on tool sophistication and more on how teachers enact technology in service of clear mathematical purposes. The review proposes an integrated pedagogical framework (mathematical purpose; instrumental orchestration; representational congruence; epistemic authority; feedback calibration; equity by design) and identifies generative AI as a multiplier of unresolved pedagogical questions rather than a straightforward solution. Priority research directions are longer classroom-embedded trials, process-sensitive outcomes, and experiments that treat teacher orchestration as an explicit, manipulated variable.

Key Points

• Evidence quality and heterogeneity
  • Aggregate effect estimates are dominated by short, small, researcher-implemented studies; these conditions also show the largest reported effects.
  • Large-scale correlational findings (international assessments) that link school technology use to lower math performance cannot be interpreted causally because usage measures conflate remediation, compensation, and different pedagogies.
• Mechanism matters more than gadgetry
  • Tool-class comparisons indicate that impact depends on the epistemic function a tool performs (e.g., sense-making, practice, assessment), not the technical sophistication or novelty of the tool.
• Teacher enactment is decisive
  • Availability of digital tools rarely changes classroom practice; teacher orchestration and enactment are the principal determinants of whether digital mediation alters mathematical activity and learning.
• Integrated pedagogical framework (core elements)
  • Mathematical purpose: align tech use to precise learning objectives.
  • Instrumental orchestration: teacher-led structuring of interactions with tools.
  • Representational congruence: match between tool representations and target mathematical concepts.
  • Epistemic authority: who/what is producing and validating mathematical justification (heightened issue with generative AI).
  • Feedback calibration: timing, specificity and pedagogical tuning of feedback loops.
  • Equity by design: anticipate and mitigate uneven access and differential instructional impacts.
• Generative AI
  • Relocates production of mathematical justification (from student/teacher to model), amplifying questions about epistemic authority, assessment validity, and the roles of scaffolding and teacher mediation.
• Research gaps and priorities
  • Need for longer, classroom-embedded trials; process-sensitive and proximal outcome measures; and studies that explicitly manipulate teacher orchestration rather than treating it as background variation.

Data & Methods

• Literature identification
  • Structured searches of open scholarly databases and indexes, supplemented by backward and forward citation searching, and review of authoritative institutional reports.
• Appraisal criteria
  • Each study assessed for design adequacy (experimental vs. observational), intervention duration, validity of outcome measures (construct and proximal vs distal outcomes), sample representativeness, and consistency with independent findings.
• Synthesis approach
  • Thematic synthesis across studies (not a study-by-study meta-analysis); emphasis on patterns by study design, implementation context, and tool epistemic function.
• Limitations of the evidence base
  • Overrepresentation of short-term, small, researcher-controlled trials; scarcity of long-term, teacher-implemented classroom trials and process-tracing measures that reveal mechanisms.

Implications for AI Economics

• Economic evaluation and cost-effectiveness
  • Simple procurement decisions based on tool features or short-term pilot effects are likely misleading. Economic appraisal should value teacher training/orchestration and longer-run, embedded implementation when estimating cost-effectiveness of edtech/AI interventions.
• Labor markets and complementarities
  • Teacher enactment remains central: generative AI is more likely to complement teaching when policy and incentives focus on teacher capabilities (orchestration, assessment calibration) than to substitute for them. Models of technology adoption should treat teacher skill and pedagogical change as key complementarities.
• Public procurement and market design
  • Evidence recommends procurement practices that fund implementation support (professional development, classroom coaching) and require rigorous, classroom-embedded evaluations rather than purchasing based on vendor claims or short pilots.
• Measurement and evaluation design for economists
  • Use cluster RCTs, stepped-wedge designs, or factorial trials that manipulate teacher-level supports (coaching, scripted orchestration, assessment integration) separately from tool provision. Include process measures, short-term proximal outcomes, and long-term attainment/ labor-market signals.
• Signaling, assessment validity, and credentialing
  • Generative AI changes who produces mathematical justification, complicating assessment and signaling mechanisms that connect schooling to labor-market outcomes. Economists should study how AI-mediated assessment affects the reliability of grades, credentials, and human capital signaling.
• Equity and distributional concerns
  • Equity-by-design is essential: market deployments without attention to differential access, differential teacher capacity, or unintended remediation/compensation uses can exacerbate inequalities. Cost–benefit analyses should incorporate distributional weights and the costs of remediation or targeted supports.
• Research agenda for AI economists
  • Estimate returns to investments in teacher orchestration (PD/coaching) vs. tool subsidies.
  • Structural or reduced-form models of adoption that include teacher heterogeneity and school incentives.
  • Longitudinal studies tying classroom technology use (and modes of orchestration) to later outcomes (STEM enrollment, labor-market returns).
  • Experiments that vary epistemic authority cues (e.g., AI as answer-provider vs. AI as explanation-support) to measure effects on learning, assessment reliability, and signaling.
• Policy implications
  • Fund and require rigorous, implementation-focused evaluations for AI/edtech procurement.
  • Prioritize investments that build teacher capacity to orchestrate technology and calibrate feedback rather than buying more sophisticated tools alone.
  • Monitor and regulate assessment use of generative AI to preserve validity of credentials and labor-market signaling.

If useful, I can draft specific experimental designs (cluster RCTs or factorial trials) or a simple economic model that formalizes teacher–technology complementarities and policy trade-offs.