Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing

Gender bias in LLMs has been studied extensively in model outputs, with biased prompts shown to amplify stereotyped generations. Whether such bias propagates into text produced by humans who use these systems, however, remains underexplored. We investigate whether gender bias in an LLM writing assistant transfers into career plan essays written by students. We first verify that a gender-biased prompt induces gender-differentiated language in LLM-generated essays, while a neutral prompt does not. We then recruited participants (N = 123) in a controlled environment to write career plan essays for paired biographical profiles differing only in gender under three conditions: no AI assistance, neutral LLM assistance, or gender-biased LLM assistance. Students in the biased condition produced essays with a significantly larger agentic gap and more gender-stereotypic occupation suggestions than those in the control and neutral conditions. Our results also reveal that this bias transfer is asymmetric: agency is suppressed in female-target essays while male-target writing remains largely unaffected. Our findings highlight the risk of bias propagation in AI-assisted writing, calling for fairness-aware design in educational AI tools.

Summary

Main Finding

Exposure to a gender-stereotype–inducing system prompt in a writing assistant transfers measurable gender bias into student-authored career-plan essays. Students who collaborated with a gender-biased LLM produced a larger agentic gap between male- and female-target essays and made more gender-stereotypic occupational recommendations than students in either a neutral-LLM or no-AI control condition. The transfer is asymmetric: bias appears primarily as suppression of agentic language for female targets rather than elevation for male targets. A carefully designed neutral prompt avoided bias transfer and even reduced baseline stereotyping.

Key Points

Experimental overview
- Between-subjects lab experiment with three conditions: No AI (control), Neutral-LLM assistance, Biased-LLM assistance.
- Participants: N = 123 undergraduates (mean age 23.1; 89 male, 34 female) each wrote one career-plan essay for a provided biography (gender of biography randomized).
- LLM stimulus validation: 1,600 LLM-generated essays (400 male/400 female × 2 prompt variants) were used to verify that the biased prompt produced gender-differentiated outputs while the neutral prompt did not.
Measures
- Agentic language gap (Δagency): difference in mean agency score (proportion of sentences classified as agentic) between male- and female-target essays.
- Stereotype Congruence Rate (SCR): proportion of essays whose recommended occupation matches population-level gender-majority occupations (BLS CPSAAT11), with >60% threshold used to label stereotyped occupations.
- Agency classifier: BERT-based model (Wan & Chang 2025) fine-tuned on LAC dataset (91.69% test accuracy).
Key quantitative results
- LLM validation: biased LLM produced a large agentic gap (Δ = 0.343, Cohen’s d = 1.903, p < .001) and high SCR (0.740); neutral LLM had a small/non-significant gap (Δ = 0.019) and lower SCR (0.351).
- Human participants:
  - Biased condition: largest agentic gap (Δ = +0.242, d = 1.510, p < .001).
  - Neutral condition: near-zero/non-significant gap (Δ = +0.032).
  - Control (no AI): intermediate but significant gap (Δ = +0.117, p = .045).
- Asymmetry: both LLM outputs and participant essays show female-target agency sharply reduced under biased prompts (LLM female agency dropped from 0.591 to 0.288 — a 51% relative reduction), while male-target agency remained roughly stable.
- Neutral LLM: not only avoided creating bias but produced the lowest rate of gender-stereotypic occupational recommendations across the three conditions.
Robustness checks
- SCR labeling thresholds (55%, 60%, 65%) tested; biased condition produced highest SCR in all cases.
- Occupation matching used fuzzy matching to BLS taxonomy with manual resolution for ambiguous cases.

Data & Methods

Stimuli and task
- Four balanced biography pairs (only name/pronouns varied) describing undergraduate subjects; participants instructed to write ≥200-word career plans addressing career choice, strengths, preferred work style, and anticipated challenges.
LLM configuration
- Model: llama-3.3-70b-instruct, temperature = 1.0.
- Two system prompts:
  - Neutral: instructs career recommendations be based on subject’s experiences/skills only (no gender cues).
  - Biased: introduces gender-stereotypic associations, steering model toward male-typical occupations for male subjects and female-typical for female subjects.
- LLM-generated corpus: 100 generations per biography name list (100 male names, 100 female names) → total 1,600 LLM essays for validation.
Human study procedure
- Controlled lab environment to block external AI use.
- In LLM conditions, participants interacted via a custom UI; required minimum of 3 model prompts, max 15 (API constraints).
- Pre- and post-surveys collected (demographics, AI familiarity, perceived influence).
- Debriefing revealed gender-bias focus only at the end.
Outcome coding
- Agency: sentence-level classification into agentic vs communal; per-essay agency = proportion agentic.
- Occupation extraction: primary source was essay text, cross-checked with post-survey answer; mapped to BLS CPSAAT11 categories using fuzzy-match (RapidFuzz threshold ≥85) and manual review.
- SCR: count of essays where the recommended occupation’s workforce majority (>60%) aligns with the biography subject’s gender.
Statistical tests
- Welch’s t-tests on per-essay agency scores; Cohen’s d effect sizes.
- Fisher’s exact tests for SCR comparisons.
- Two-way ANOVA for Gender × Condition interactions (paper reports a significant interaction).
Limitations noted by authors (important for interpretation)
- Participant sample skewed male (72%), single university, English-medium undergraduates nearing graduation — limits generalizability.
- Lab task is an acute exposure; long-term effects or repeated-exposure dynamics remain untested.
- Manipulation confined to prompt-level changes (no model weight changes) and uses one open-source model; behaviour may differ across models and deployment contexts.

Implications for AI Economics

Market externalities and human capital formation
- AI writing assistants can create negative externalities by subtly shaping the content of students’ career narratives and recommendations, which may influence self-concept, signaling, and peer perceptions—potentially altering occupational choices and labor supply into stereotype-aligned fields.
- If repeated at scale (given high student adoption rates), small per-user changes in expressed agency or occupational framing could aggregate into measurable shifts in enrollment patterns, career advice flows, and ultimately occupational gender composition — reinforcing occupational segregation and affecting labor-market outcomes and productivity.
Distributional effects and inequality
- Asymmetric suppression of female agency implies unequal downstream effects: women’s expressed qualifications and aspirations could be systematically under-emphasized when assisted by biased AI, exacerbating gender gaps in high-status, high-pay occupations. This raises equity concerns and potential long-term income inequality implications.
Product design and firm strategy
- Firms embedding LLM assistants in educational products face reputational, regulatory, and economic risks if those tools propagate stereotype-consistent outputs. There is a clear commercial incentive to design neutral or parity-promoting prompts and to validate models using metrics like agentic gap and SCR.
- Neutral prompt configurations can reduce stereotyping and may be marketed as a value-added fairness feature — an avenue for product differentiation.
Policy and regulation
- Findings motivate regulation or procurement standards for educational AI: regular bias audits (e.g., agentic gap, SCR), transparency about system prompts and bias tests, and minimum fairness performance for tools deployed in schools/universities.
- Given potential aggregate externalities, public-sector intervention (guidance, subsidies for debiasing, requirements for disclosure) may be warranted to correct market failures where private incentives underestimate social harms.
Measurement and economic modeling opportunities
- The paper provides operationalizable metrics (agentic gap, SCR tied to BLS occupational shares) that can be incorporated into economic models estimating the impact of AI-mediated framing on occupational choice and labor-market flows.
- Next steps for applied work: longitudinal field studies linking AI-assisted writing exposure to actual labor-market outcomes (majors, internships, job applications), and structural models to quantify long-run effects on occupational gender composition and wages.
Cost-benefit and deployment trade-offs
- Deploying neutral/debiased assistants likely entails development and validation costs but could avoid downstream social costs (reduced diversity in occupations, litigation/reputational costs). Economists and decision-makers should evaluate these trade-offs: per-user debiasing cost versus estimated social cost of increased occupational segregation.
Interventions suggested (actionable for policymakers and firms)
- Require/encourage pre-deployment bias validation using real-world tasks and metrics like agentic gap and SCR.
- Offer default neutral prompts and transparent controls allowing instructors/users to select anti-stereotype settings.
- Educator guidance and AI literacy: teach students about potential model influence and encourage active revision rather than blind acceptance of suggestions.
- Continuous monitoring: log interactions and periodically re-run bias audits as models and prompts evolve.

Overall, the study shows that prompt-level biases in widely used LLM assistants can propagate into human-authored content in ways that may have meaningful economic and distributional consequences. Incorporating fairness checks into product design, procurement, and policy can help mitigate these risks before they scale.

Assessment

Paper Typerct Evidence Strengthhigh — A randomized design provides credible causal identification of the effect of biased LLM assistance on human-written essays, and the authors validated the LLM manipulation in a pretest; effect heterogeneity (asymmetry by target gender) is documented statistically. Methods Rigormedium — Strong experimental design and pretesting, but limitations include modest sample size (N=123), a lab/education-student sample, likely use of a single model and prompt variants, potential subjectivity in coding agentic language and occupation suggestions, and limited detail on blinding and robustness checks. SampleControlled experiment with 123 student participants who each wrote career-plan essays for paired biographical profiles differing only by gender; participants were assigned to one of three conditions (no AI, neutral LLM, gender-biased LLM); the study also includes a pretest showing the biased prompt induces gendered LLM output. Themeshuman_ai_collab inequality IdentificationRandomized controlled experiment: participants (N=123 students) were randomly assigned to three conditions (no AI, neutral LLM assistance, gender-biased LLM assistance) and wrote career-plan essays for paired biographical profiles that differed only by target gender; causal effects identified by between-group differences backed by a pretest verifying that the biased prompt produced gender-differentiated LLM output. GeneralizabilityStudent (likely college) sample — not representative of general adult population or working professionals, Artificial/lab writing task — may differ from real-world, high-stakes career counseling or workplace writing, Single or limited set of LLM prompts/models — results may not generalize across models, prompt styles, or updated models, Short-term, immediate-writing effects only — no evidence on long-term beliefs, career choices, or labor-market outcomes, Cultural/geographic context and participant demographics not specified — limited cross-cultural generalizability, Measurement relies on coded language and occupation suggestions, which may involve subjective judgment

Claims (7)

Claim	Direction	Outcome	Confidence & Evidence	Details
A gender-biased prompt induces gender-differentiated language in LLM-generated essays. Output Quality	positive	gender-differentiated language in LLM-generated essays	Reading fidelity high Study strength medium	not reported 0.6
A neutral prompt does not induce gender-differentiated language in LLM-generated essays. Output Quality	null_result	gender-differentiated language in LLM-generated essays	Reading fidelity high Study strength medium	not reported 0.6
The study recruited participants (N = 123) in a controlled environment to write career plan essays for paired biographical profiles differing only in gender under three conditions: no AI assistance, neutral LLM assistance, or gender-biased LLM assistance. Other	null_result	experimental sample and conditions (study design)	Reading fidelity high Study strength high	n=123 1.0
Students in the gender-biased condition produced essays with a significantly larger agentic gap than those in the control and neutral conditions. Output Quality	positive	agentic gap (difference in agentic language between male- and female-target essays)	Reading fidelity high Study strength medium	n=123 0.6
Students in the gender-biased condition produced more gender-stereotypic occupation suggestions than those in the control and neutral conditions. Output Quality	positive	frequency of gender-stereotypic occupation suggestions	Reading fidelity high Study strength medium	n=123 0.6
Bias transfer from the LLM is asymmetric: agency is suppressed in female-target essays while male-target writing remains largely unaffected. Output Quality	mixed	agency in essays by target gender (suppression in female-target essays, no change in male-target essays)	Reading fidelity high Study strength medium	n=123 0.6
These findings highlight the risk of bias propagation in AI-assisted writing, calling for fairness-aware design in educational AI tools. Governance And Regulation	negative	risk of bias propagation and need for fairness-aware design	Reading fidelity high Study strength speculative	n=123 0.1

Biased writing assistants nudge students toward stereotypical career descriptions, cutting expressions of agency in essays about women. Neutral prompts avoid this effect, and male-target essays remain largely unaffected, revealing asymmetric bias transfer.