AI-generated STEM instruction favors privileged profiles: audits of four large language models reveal up to a 2.55-grade-level advantage for the most privileged over the most marginalized, with income, medium of instruction and disability driving the largest single effects and biases persisting even within elite institutions.

Main Finding

LLM-generated STEM instructional content systematically disadvantages marginalized student profiles across both Indian and U.S. educational contexts. Income is a consistent and significant predictor of simpler treatment in every model and context; in India, medium of instruction (English vs. Hindi/regional) produces the largest single effect. Biases compound non‑additively across intersecting identities (caste/race, income, location, disability, institution tier), producing gaps up to 2.55 grade levels between the most privileged and most marginalized synthetic profiles. These patterns hold across four different LLMs (Qwen 2.5-32B‑Instruct, GPT‑4o, GPT‑4o‑mini, GPT‑OSS‑20B) and across selection/role framing, implying structural origins in training/alignment rather than idiosyncratic model choices.

Key Points

• Audited models: Qwen 2.5‑32B‑Instruct, GPT‑4o, GPT‑4o‑mini, GPT‑OSS‑20B (temperature = 0).
• Two contexts: India (caste, medium of instruction, college tier, board, location, income, gender, disability) and U.S. (race/ethnicity, HBCU attendance, school type, college tier, location, income, gender, disability).
• Two tasks:
  • Ranking task → Mean Choice Value (MCV): model’s selected difficulty level (1–5).
  • Generation task → Mean Grade Level (MGL): average of Flesch‑Kincaid, Gunning Fog, Coleman‑Liau readability scores.
• Datasets: MATH‑50 (general math benchmark) and JEEBench (India engineering exam problems).
• Sample design: stratified sampling of 100 synthetic intersectional profiles per context (coverage across 5,184 Indian and 4,860 U.S. possible combinations).
• Bias metrics: Mean Absolute Bias (MAB) and Maximum Difference Bias (MDB); SHAP used for feature attribution; statistical tests with FDR correction and Cohen’s d reported.
• Quantitative highlights:
  • Intersectional gap (most privileged vs most marginalized): up to 2.55 grade levels.
  • Income effect sizes across conditions: Cohen’s d ≈ 0.21 to 0.81.
  • In India, English‑medium profiles received the highest complexity outputs (authors report 100% of highest‑complexity outputs assigned to English‑medium profiles in their tests).
  • Disability cues trigger systematically simpler explanations; effect nearly doubles when the model is asked to adopt a student (rather than teacher) perspective.
  • Bias persists inside elite institutions (e.g., low‑income / caste‑oppressed IIT profiles ≈ 0.9 grade levels below privileged IIT peers; similar penalties found for low‑income rural students at Ivy tier).
• Consistency: Direction and broad magnitude of biases are comparable across all four model architectures and both tasks; variation mainly in consistency of application.

Data & Methods

• Profile construction:
  • India: 8 attributes (caste: General/OBC/SC/ST; college tier: IIT/NIT/State Gov/Private; location: Metro/Tier‑2/Rural; medium: English/Hindi/regional; board: CBSE/ICSE/State; gender; income: High/Mid/Low; disability).
  • U.S.: 7 attributes (race/ethnicity categories; college tier including Ivy/HBCU etc.; location: Urban/Suburban/Rural; school type: Public/Private/Charter; gender; income; disability).
  • Stratified sampling produced 100 profiles per context to balance representation.
• Tasks and samples:
  • Ranking experiments: 100 profiles × 7 subjects × 2 role frames (teacher/student) → 1,400 ranking trials reported.
  • Generation experiments: MATH‑50: 100 profiles × 7 subjects × 3 problems → 2,100 generations. JEEBench: 50 problems sampled → 5,000 generations reported.
• Complexity measures:
  • MCV (ranking) and MGL (generation, average of three readability indices).
• Bias quantification:
  • Normalize MCV/MGL within subgroup and subject.
  • MAB = average absolute deviation from subgroup mean; MDB = max difference between subgroup extremes.
  • SHAP applied to decompose dimension contributions to complexity differences.
  • Statistical validation: t‑tests, Cohen’s d, KL divergence; p‑values FDR‑corrected.
• Models and settings:
  • Two closed models (GPT‑4o, GPT‑4o‑mini) and two open‑weight models (Qwen 2.5‑32B‑Instruct, GPT‑OSS‑20B).
  • Deterministic decoding (temperature = 0); served with float16 for open models.

Implications for AI Economics

• Human capital formation and inequality
  • Differential instructional complexity translates into differential learning inputs. If marginalized students systematically receive simpler content, their skill accumulation and human capital growth may be reduced—potentially lowering lifetime earnings and perpetuating intergenerational inequality.
  • Non‑additive intersectional penalties (up to 2.55 grade levels) imply compounding losses that standard single‑axis evaluations underestimate; these can widen returns‑to‑education gaps across population groups.
• Labor market and signaling effects
  • Reduced exposure to complex explanations could impair readiness for advanced STEM training, reducing the supply of qualified candidates from marginalized groups and reinforcing signaling advantages of privileged students/institutions.
  • Employers and institutions relying on LLM‑assisted preparation may inadvertently favor candidates from groups that received richer LLM tutoring.
• Aggregate productivity and growth
  • If widely deployed LLM tutors systematically under‑educate marginalized cohorts, aggregate productivity gains from AI in education may be uneven, lowering aggregate social returns and potentially increasing macroeconomic inequality.
  • Economic growth models that incorporate human capital accumulation should account for AI‑mediated, distributional learning effects; neglecting them risks overstating egalitarian gains from AI in education.
• Market and regulatory responses
  • Market: demand for localized/fairness‑tuned models (language, caste/race‑aware training) will rise; institutions may procure models audited for intersectional equity as a competitive or compliance signal.
  • Policy: audits that are intersectional and cross‑cultural should be required pre‑deployment. Disclosure requirements around personalization behavior, language/region training coverage, and subgroup performance should be mandated.
  • Cost/benefit: regulators and funders should evaluate both efficiency gains from LLM assistance and distributional harms; remediation (fine‑tuning, localized data, targeted tutoring supplements) entails nontrivial costs but may be necessary to prevent widening inequality.
• Design & intervention economics
  • Intervention options (and their economic tradeoffs):
    • Intersectional audits and monitoring (low marginal cost, early detection).
    • Targeted fine‑tuning / data augmentation for underrepresented languages and intersectional profiles (higher upfront cost; reduces biased priors).
    • Prompt engineering and guardrails to equalize complexity conditional on demonstrated ability rather than demographic cues (operationally cheaper but brittle).
    • Subsidized deployment of fairness‑tuned models to underserved institutions (public expenditure; direct redistribution of learning inputs).
  • Comparative-effectiveness evaluation recommended: run randomized trials to estimate causal impacts of bias‑mitigating interventions on learning outcomes and downstream earnings to guide resource allocation.
• Measurement and modelling recommendations for economists
  • Treat LLM personalization as a channel that shifts effective instructional quality; parameterize differential learning rates by subgroup in human capital accumulation models.
  • Quantify welfare impacts using counterfactual scenarios: (i) baseline no‑LLM, (ii) LLM with bias, (iii) LLM with mitigation—estimate present value of lifetime earnings and distributional metrics (Gini, poverty headcount).
  • Incorporate institution‑level heterogeneity: model interactions between institutional prestige and LLM effects (since biases persist inside elites).
  • Track language composition and training source exposure as key determinants of distributional outcomes across countries.
• Policy takeaways
  • Require intersectional, cross‑cultural audits before large‑scale educational deployment.
  • Fund and prioritize training data and alignment efforts for non‑dominant languages and marginalized contexts.
  • Mandate transparency and reporting on subgroup performance metrics (MCV/MGL analogues) for educational LLMs.
  • Support remediation (targeted tutoring, subsidies) where audits reveal systematic harms.

Caveats and limits (affect interpretation and economic projections) - Synthetic profiles approximate but do not equal real student behavior; downstream educational outcomes were not measured—readability and selected difficulty are proxies for instructional quality. - Datasets focus on STEM/engineering; generalization to other subjects or K–12 contexts requires further study. - Only four models tested; while architectures varied, broader model family coverage would strengthen claims about universality. - Deterministic decoding (temperature = 0) removes stochastic variation but may not reflect all real‑world deployments.

Overall, the paper demonstrates measurable, consistent, and economically consequential patterns of intersectional disadvantage in LLM‑generated STEM instruction. For economists and policymakers, these results argue for integrating fairness audits into cost‑benefit analyses of AI in education and for modeling the distributional (not just aggregate) impacts of AI‑mediated learning.