A new occupation-grounded benchmark finds leading LLMs correctly answer roughly 60% of profession-specific, source-verifiable questions, but performance is highly uneven — healthcare questions score near 78% while many trade and enforcement occupations score near zero, and model rankings remain stable across formats.

Main Finding

ORQA introduces a scalable, source-traceable benchmark for occupation-level knowledge in LLMs and shows substantial heterogeneity in model performance across occupations. Using 480 human-verified, evidence-backed questions drawn from 187 authoritative websites covering 116 occupations (all 21 SOC major groups), the authors find frontier models (Claude Opus 4.6, GPT‑5.4, Claude Sonnet 4.6) cluster near ~58–62% on closed-book multiple choice while smaller/open-weight models score ~33–41%. Performance varies widely by occupation (e.g., healthcare ≈78% vs. office/admin ≈40%; some occupations score ~0%). ORQA correlates strongly with other economic model-evaluation signals (Spearman ρ ≈ +0.90 with GDPval).

Key Points

• ORQA (Occupation Related Question and Answer) is designed to test source-verifiable, occupation-specific factual/procedural knowledge (reporting responsibilities, thresholds, procedures, guidelines), not general problem-solving or end-to-end task execution.
• Dataset: 480 high-quality items, 116 occupations, 187 source hosts, covers all 21 SOC major groups.
• Benchmark format: closed-book 6-choice multiple choice (includes “All of the above” and “None of the above”) and an open-ended free-response evaluation scored by human judges. Closed-book evaluation uses 3 random seeds per question.
• Top model results (closed-book MC):
  • Claude Opus 4.6 ≈ 62.4%
  • GPT‑5.4 ≈ 60.3%
  • Claude Sonnet 4.6 ≈ 58.7%
  • Mid-tier and older/smaller models: 33–48%
• Open-ended results follow the same ordering but with lower absolute scores (GPT‑5.4 ≈ 54.2%, Opus ≈ 53.1%, Sonnet ≈ 50.6%).
• Performance is highly occupation-dependent: some occupations have near-perfect scores for some models while others are essentially unsolved (e.g., Sheet Metal Workers, Fish and Game Wardens).
• Wage-bill weighting and open-ended vs. multiple-choice do not materially change the model ranking.
• ORQA shows high correlation with GDPval (Spearman ≈ +0.90) and GDPval-AA Elo (≈ +0.92), indicating alignment with other economic benchmarking signals.
• Human curation: a hybrid pipeline combining automated generation and human review; automated “good” vs. human agreement ≈ 67%.

Data & Methods

• Base occupations & weighting: uses BLS May 2024 Occupational Employment and Wage Statistics and O*NET mapping; items allocated across SOC major groups by wage bill share.
• Source selection: job-specific whitelists of authoritative publishers (regulators, licensing bodies, professional orgs, government publications). No documents are used from domains outside the whitelist; academic/research papers are filtered out.
• Document processing:
  • Fetch documents constrained to job-specific whitelists.
  • Chunking (2.5–4 KB windows, overlapping) and evidence-card extraction using GPT-4o to produce contiguous source quotes and metadata (task/procedure, occupation).
• Item generation:
  • Each evidence card becomes a 6-choice MCQ; correct answer must be directly entailed by the source quote; distractors are drawn from the same document.
  • Constraints: similar answer lengths, no obvious hints, inclusion of “All/None of the above,” only one correct choice.
• Automated quality controls:
  • Checks include entailment test, semantic plausibility of distractors, lack of paraphrase overlap, non-solvability by three baseline LLMs in parallel (filter too-easy items), and tests to prevent trivially answerable items via general knowledge.
  • Failed items go through regeneration attempts; final step mirrors human “good” standard.
• Human review: two annotators manually checked items, sources, and answer options against quality standards.
• Evaluation: 15 models tested closed-book (no retrieval/browsing); closed-book MC with three seeds; open-ended responses judged by a three-judge panel. Cluster bootstrap used for uncertainty; also report wage-bill weighted aggregates.
• Public release: data and code available (orqabench.org; reproducibility code repo linked).

Implications for AI Economics

• Occupation-specific model assessment: ORQA provides a scalable, verifiable way to measure whether LLMs know occupationally relevant, source-backed facts—useful for firms, regulators, and researchers assessing model readiness for particular job roles.
• Heterogeneous labor impact: large cross-occupation and cross-model performance differences imply uneven potential for productivity gains and automation risks across the labor market. This heterogeneity should be incorporated in exposure/adoption and labor-displacement analyses rather than relying on aggregate capability metrics.
• Procurement and deployment decisions: employers can use occupation-indexed benchmarks like ORQA to choose models tailored to specific occupational knowledge needs (e.g., healthcare vs. maintenance).
• Policy and regulation: source-traceable testing aligns with regulatory and compliance needs (e.g., licensing, safety-critical guidance). ORQA-style benchmarks could inform standards for model certification in regulated occupations.
• Measurement for economic studies: wage-bill weighting and SOC coverage make ORQA usable as an input into empirical studies of AI adoption and productivity that need occupation-level model performance measures; strong correlation with GDPval suggests it can complement existing exposure metrics.
• Cautions and limitations for applied use:
  • ORQA assesses closed-book factual/procedural knowledge drawn from authoritative texts; real-world systems often include retrieval, tool use, or human-AI collaboration—so ORQA scores are a lower-bound on deployed effectiveness.
  • Coverage limited to occupations with authoritative online documentation; not all jobs or tacit skills are captured.
  • Dataset size (480 items) is moderate; useful for comparison but not exhaustive per occupation.
  • The pipeline excludes academic research sources by design, which suits accountability-focused use cases but omits research-derived knowledge that may be relevant in some roles.

Overall, ORQA offers a practical, scalable method to map LLM knowledge onto occupation-relevant, source-verifiable questions. For economic research and policy it provides a complementary metric to exposure/adoption studies, enabling more granular assessments of where AI may be practically useful or risky in the workforce.