A large language model ranks freelancer–project fit largely by skills and experience but also reads extra signals into profiles; while average bias against minority groups is small, intersectional differences mean productivity cues are weighted differently across demographic subgroups.

Main Finding

A large general-purpose LLM (Gemini 2.0 Flash), when placed in the role of a recruiter and evaluated on a fully factorial set of synthetic freelancer profiles and project briefs from a European freelance platform, largely behaves like an economic hiring evaluator: it places strongest weight on productivity signals (experience, reputation, skill match) and much weaker average weight on socio‑demographic attributes. However, intersectional heterogeneity shows the model uses demographic cues to modulate how it interprets productivity signals, creating subtle channels for unequal treatment even when average discrimination is small.

Key Points

• Research questions addressed:
  • RQ1: Which profile/job attributes does the LLM emphasize or ignore?
  • RQ2: Does the LLM’s decision logic vary across socio‑demographic groups?
  • RQ3: How to compare LLM and human recruiter decision logic experimentally?
• Experimental scale and outputs:
  • 10,800 synthetic freelancer profiles × 16 synthetic briefs → 172,800 profile–brief pairs.
  • Gemini 2.0 Flash provided hiring probabilities on a 10‑point scale; each pair scored 3 times and averaged.
  • Score distribution: mean ≈ 6.5, SD ≈ 1.3, observed range ~3–9.8; only 2.48% of pairs varied across reruns (max ±1 point).
• Attribute importance (OLS summary of implicit weights; coefficients interpreted as causal marginal effects due to fully randomized factorial design):
  • Largest penalties/rewards: Experience (largest effect magnitude, e.g., underqualified penalized ≈ −2.25 pts), Reputation (≈ −1.19 pts for low reputation), Skill match (≈ −0.69 pts for distant mismatch).
  • Other notable effects: Daily rate (≈ −0.52), Remote preference (≈ −0.50). Part‑time, firm size, industry, education had small effects.
  • Socio‑demographic attributes (perceived gender/ethnicity, education) had minimal average direct effects (ranked low), but heterogeneity analyses show group‑dependent weighting of productivity signals (intersectional effects).
• Fairness and alignment:
  • On average, minimal direct discrimination by gender/ethnicity in the model’s scores.
  • Intersectional differences: the same productivity signal (e.g., experience, reputation) can affect scores differently depending on perceived demographic attributes — a subtler, context‑dependent source of unequal outcomes.
• Methodological contribution:
  • A fully factorial synthetic design adapted from correspondence studies enables causal attribution of LLM decision weights and can be reused to compare LLMs with human recruiters using identical stimuli.

Data & Methods

• Context and scope:
  • Data and attribute distributions are drawn from a large European freelance marketplace.
  • Primary occupational focus: full‑stack developers in France (high male name prevalence); robustness checks on SEO content writers reported in appendix.
• Synthetic data construction:
  • Full factorial design for profiles and briefs to break correlations and enable causal identification.
  • Profiles (10,800): varied on skills (exact / close substitute / distant), experience (1 / 5 / 9 years), work arrangement preferences (remote/on‑site, full/part‑time), reputation (projects 0/1/5; rating 0/5; badge yes/no), daily rate (300–500€), prior employer (SME vs large; e‑commerce vs banking), and socio‑demographics signaled by first names (European male, European female, Arabic male); education (BSc vs MSc).
  • Briefs (16): varied recruiter gender (name), firm size (SME vs large), remote vs onsite, full vs part time; some dimensions held constant (e.g., required duration, tech stack, required experience).
• LLM elicitation:
  • Model: Gemini 2.0 Flash, prompts in French.
  • Prompt asked for a hiring probability (10‑point scale) and a brief explanation of reasoning; no fairness correction instruction.
  • Each profile–brief pair scored three times; mean used.
• Statistical analysis:
  • OLS regressions of average LLM score on binary indicators for attribute levels; standard errors clustered at brief level.
  • Given orthogonalized design, OLS coefficients interpreted as marginal causal effects.
  • Reported model summary: R^2 ≈ 0.90 — linear additive approximation explains most score variation in this experimental context.
• Limitations noted by authors:
  • Full factorial design trades external representativeness for causal identification.
  • Prompt framing (hiring probability) and profile formatting may influence model reasoning; results may depend on prompt, language, or model family.
  • Single LLM and occupational focus limit generalizability; real‑world downstream outcomes not observed.

Implications for AI Economics

• Theoretical alignment: LLMs can internalize economic signaling logic. In this experiment the LLM rewarded close matches and trust signals (experience, reputation) in ways consistent with labor‑economics models of signaling and productivity inference.
• Measurement & auditing tools: Fully factorial synthetic designs adapted from correspondence studies provide a rigorous, causal way to estimate implicit attribute weights in LLM decision‑making and to surface interaction (intersectional) effects that simple attribution methods might miss.
• Policy and platform implications:
  • Even when average demographic effects are small, intersectional heterogeneity can produce unequal outcomes; audits should measure both average and subgroup‑conditional effects.
  • Platforms using LLMs for candidate ranking or screening should monitor how the model reweights productivity signals across groups and consider calibration, counterfactual testing, or constraint mechanisms to prevent unintended disparate impacts.
  • Regulatory auditing frameworks could adopt similar experimental stimuli to evaluate vendor claims about fairness and alignment.
• Research directions:
  • Direct human–AI comparison: deploy the identical synthetic stimuli to human recruiters to quantify alignment/misalignment and the welfare implications of replacing or assisting human screening with LLMs.
  • External validity tests: evaluate other LLMs, prompt framings, languages, and occupational contexts; link model scores to downstream hiring outcomes to measure market‑level effects.
  • Dynamic and systemic effects: study how LLM recommendations change demand‑side behavior (which candidates are contacted) and feedback loops that could amplify inequalities over time.
• Practical takeaway for economists and practitioners: LLMs can be powerful, interpretable evaluators of unstructured candidate information, but rigorous causal audits (including intersectional analyses) are necessary before large‑scale deployment in hiring to detect subtle disparate‑impact mechanisms that simple averages hide.