Large language models frequently mis-rank candidates on a controlled resume dataset, failing to reliably pick objectively stronger applicants and inconsistently favoring different demographic groups; the findings expose gaps in validity and auditing tools for LLM-based resume screeners.

Main Finding

The paper introduces a reproducible framework to measure the validity (criterion and discriminant) of LLM-based resume screeners by constructing novel resume pairs with a known ground truth. Using that framework, the authors find many off-the-shelf LLMs fail to reliably (a) pick the demonstrably more-qualified candidate and (b) abstain or remain indifferent when candidates are equally qualified. Larger/newer models perform better but are not perfect; most models’ errors are dominated by unjustified abstention, and selection rates across demographic groups can be uneven (sometimes favoring historically marginalized groups).

Key Points

• Problem addressed: Prior work emphasizes fairness metrics (selection parity) but often neglects validity—whether models actually identify superior candidates based on relevant qualifications.
• Evaluation framework:
  • Constructs resume pairs with clear ground truth using job descriptions (required vs. preferred qualifications).
  • Creates (1) unequal pairs where one resume strictly dominates on relevant qualifications and (2) equal pairs that differ only in irrelevant demographic signals.
  • Measures model behavior in pairwise comparisons, including capability to abstain.
• Metrics defined:
  • CriterionValidity: fraction of unequal pairs where the model picks the more-qualified resume.
  • DiscriminantValidity: fraction of equal pairs where the model abstains (i.e., recognizes indifference).
  • UnjustifiedSelection / UnjustifiedAbstention: error-type breakdown.
  • OverAssessment: rate a demographic group’s candidates are preferred when unqualified.
  • Forced-choice selection rates: fairness when abstention is disallowed.
• Main empirical patterns:
  • Model performance generally improves with scale/newer releases, but many models do not exceed high validity thresholds (e.g., 0.95).
  • Example results: Claude Sonnet 4 ≈ 0.96 average CriterionValidity; GPT‑5, Gemini‑2.5‑Pro often > 0.90; Llama variants performed substantially worse (e.g., Llama‑3.1‑8B ≈ 0.64 for k=1).
  • Most errors are due to unjustified abstention rather than choosing the less-qualified candidate.
  • Discriminant validity is weak: models often fail to abstain on equally-qualified pairs, and selection rates vary across demographic signals (occasionally prioritizing historically marginalized groups—possible evidence of over-alignment).
• Robustness and contamination:
  • Framework uses live job descriptions (Greenhouse; results generalize to LinkedIn/Indeed in appendix) and LLM-generated novel resumes to avoid train-test contamination.
  • Human validation was performed on samples of generated resumes to filter generation errors.

Data & Methods

• Data sources:
  • 186 public job descriptions scraped from Greenhouse across 25 categories (pipeline generalizable to other job boards and internal postings).
  • LLMs used to parse qualifications and synthesize resumes.
  • Demographic signals appended: four groups {Black, White} × {man, woman}, with both implicit (name) and explicit demographic cues.
• Resume generation:
  • Base resume meets required qualifications exactly.
  • c+ (more-qualified): add k preferred qualifications.
  • c− (less-qualified): remove k required qualifications.
  • Equal-pair variants: reword base resume + different irrelevant demographic signals.
  • Generation employed multiple models (Gemini‑2.5‑Pro, Claude‑Sonnet‑4) to reduce generation bias.
• Evaluated models:
  • Frontier and widely used LLMs (instruction-tuned/aligned): Claude Sonnet 4, Gemini 2.0 Flash, Gemini 2.5 Pro, GPT‑5, GPT‑4o‑mini, Gemma 3‑12B, Llama‑3.1‑8B, Llama‑3.3‑70B, Deepseek Chat v3.1.
• Experimental protocol:
  • Pairwise comparison setting where the screener returns one of the two candidates or abstains (⊥). Also considered forced-choice (no abstention).
  • Multiple prompt/system variations checked for sensitivity (results stable).
  • Manual validation: inspected random subsets of generated pairs and removed flawed ones; reported minimal effect on aggregated metrics.
• Key quantitative findings (examples from reported tables):
  • CriterionValidity increases with k (number of differing relevant qualifications) and with model capability, but many models fall below 0.95 at small k.
  • Llama models performed poorly at k=1 (CriterionValidity as low as 0.44 for Llama‑3.3‑70B with certain generators).
  • Larger models (Claude‑Sonnet‑4, GPT‑5, Gemini‑2.5‑Pro) often exceeded 0.90 average CriterionValidity across tested ks.
  • Error decomposition: majority of errors were unjustified abstentions rather than choosing the less qualified candidate.

Implications for AI Economics

• Labor market matching and efficiency:
  • If LLM screeners are used out-of-the-box, imperfect validity undermines firms’ ability to rank candidates by true productivity-relevant qualifications, potentially reducing match quality and hiring efficiency.
  • Frequent unjustified abstention raises recruiting costs (more human review), changing the economics of screening automation—there is a trade-off between model conservatism and human labor costs.
• Adoption and procurement decisions:
  • Firms and public agencies relying on general-purpose LLMs should factor in validity testing and potential decreases in effective throughput or increases in review labor.
  • Economic incentives may push vendors to over-align models towards social objectives (leading to over-assessment of some groups), which can produce unintended selection distortions—affecting both fairness and firm productivity.
• Regulatory and auditing implications:
  • The proposed framework is operationally useful for independent audits (e.g., compliance with local laws requiring audits of automated hiring tools). Regulators can mandate validity reporting (CriterionValidity, abstention rates, over-assessment by group) as part of procurement or certification.
  • Transparency demands: firms should disclose abstention behavior, selection-rate splits under forced-choice, and validity audit results to assess deployment impacts.
• Externalities and redistribution:
  • Systematic selection biases—even those that prioritize marginalized groups—constitute redistributive effects that can alter labor supply outcomes and potentially generate mismatches with job requirements, with macro-level productivity implications.
  • Where models abstain more for certain jobs or qualifications, disadvantaged firms (e.g., small employers lacking review capacity) may be disproportionately harmed.
• Research and market design recommendations:
  • Require validity-first benchmarks (not only fairness parity metrics) when assessing hiring AI; economic evaluations of adoption should include costs of human-in-the-loop review triggered by abstention.
  • Vendors should offer calibrated abstention thresholds and performance guarantees (CriterionValidity at specified ks) to align incentives and enable sensible cost-benefit decisions by buyers.
  • Policymakers and procurement officers should incentivize routine independent audits using frameworks that avoid train-test contamination (generate novel test cases).
• Limitations to consider in economic analysis:
  • The paper’s framework models an idealized setting—real hiring decisions involve multi-dimensional trade-offs, noisy signals, and heterogeneous employer preferences; thus measured validity is a conservative estimate of correctness.
  • Generated resumes may not capture every aspect of real applicant pools; applicability to end-to-end hiring pipelines requires further field validation.

If you’d like, I can: - Extract more detailed per-model numbers and present a compact table for quick comparison; - Sketch an economic model estimating cost trade-offs between unjustified abstention vs. misselection for a firm using these screeners; or - Outline policy language (audit/reporting requirements) that could be proposed for regulators based on this framework.