Generative AI can handle routine, structured parts of end-to-end data science projects but struggles on judgmental tasks: models approximate entry-level data scientists on predictable work yet vary considerably and require human verification for nuanced decisions.

Main Finding

Recent generative AI models can complete end-to-end undergraduate-style data science projects at near–undergraduate quality for routine, template-following tasks, but performance varies widely across models and drops substantially on tasks requiring judgment (e.g., measurement, abstracts, correct citations). Best models approximate a good undergraduate (≈80–85%) but still require human verification.

Key Points

• Benchmark & scoring:
  • 40 anonymized undergraduate projects; 17-category rubric totaling 45 points (A-/B+ threshold = 80% = 36/45).
  • Rubric grouped into six types: Reproducibility, Presentation, Analysis, Graphs, Referencing, Writing.
• Models tested (each generated 5 projects; auto-graded):
  • Claude Opus 4.6: 38/45 (85%)
  • GPT-5.2: 37/45 (82%)
  • Gemini 3 Flash: 35/45 (78%)
  • Grok-4: 31/45 (70%)
  • Gemini 3 Pro: 27/45 (59%)
  • Llama 4: 23/45 (51%)
  • GPT-4o: 15/45 (32%)
• Performance patterns:
  • Strong on template/structured tasks: titles, author/date/repo fields, test suites, basic reproducibility artifacts.
  • Weak on judgment-heavy or context-sensitive tasks: measurement explanations, abstracts, introductions/discussions, correct BibTeX/citations, Quarto cross-references, nuanced visualizations.
  • Significant inter-model heterogeneity — recent top models cluster near undergraduate quality, older or less capable models fall well below.
• Evaluation pipeline:
  • Developed an automated LLM-as-judge grader (primary grader: Claude Sonnet 4.5) validated against human graders and also cross-checked with GPT-5.2; grading performed per rubric category and repeated for robustness.
• Caveats noted by authors:
  • Small test per model (n=5 projects) — results indicative not definitive rankings.
  • Domain limited to City of Toronto open data and Quarto/Python workflows.
  • Potential biases from using an LLM grader (provider bias concerns); authors validated to mitigate but not eliminate this risk.
  • Models were not extensively prompt-engineered to the specific datasets; guardrails were applied in generation.

Data & Methods

• Benchmark construction:
  • Source: 40 undergraduate project repositories (converted to Python where necessary), each with required scripts: 01-download.py, 02-clean.py, 03-test.py, paper.qmd (Quarto), references.bib, README, dependency lock files.
  • Two human graders scored each project on a 17-item rubric (45 points); disagreements resolved and rubric refined.
• Auto-grader development:
  • LLM-as-a-judge approach with per-category prompts to evaluate rubric items.
  • Trained/validated on the human-graded benchmark; auto-grader run three times per generated project to produce scores.
  • Analysis done in R/tidyverse; detailed rubric and appendices provided in paper.
• Model evaluation:
  • Seven contemporary LLMs used to generate five end-to-end projects each (with some generation guardrails).
  • Outputs compiled into repos and auto-graded; aggregated scores reported by category and overall.
• Reproducibility & limitations:
  • Authors disclose use of AI in coding assistance during research but reviewed all outputs.
  • Acknowledge limitations from small sample size, single domain, grader bias, and evaluation subjectivity (figures/visual taste).

Implications for AI Economics

• Task-level substitution is uneven:
  • Routine, template-driven data science tasks are plausibly automatable soon at near–entry-level quality, implying downward pressure on purely routine portions of data-science labor demand.
  • High-judgment, contextual, and domain-knowledge tasks remain complementary to humans — these tasks are likely to retain value and require human oversight.
• Productivity vs. oversight costs:
  • Firms may realize productivity gains (faster project turnarounds, reduced routine labor) but must invest in verification, auditing, and model-selection processes to manage hallucinations, citation errors, and measurement misunderstandings. These monitoring costs moderate net gains.
• Heterogeneous adoption and skill demand:
  • Variation across models implies firms must evaluate/model-test choices; adoption will vary by organization size, domain, and tolerance for risk.
  • Demand will shift toward workers who can supervise models, validate outputs, and perform high-judgment tasks (measurement design, causal inference, domain interpretation, ethics).
• Labor market dynamics:
  • Potential reallocation of tasks from junior analysts to AI-assisted workflows; combined human+AI teams likely more productive than AI alone.
  • Displacement risk concentrated on routine entry-level tasks; jobs emphasizing synthesis, interpretation, and domain judgment less exposed.
• Policy and governance:
  • Need for standards and domain-specific benchmarks to assess model readiness in applied settings (business, government, regulated sectors).
  • Regulatory attention warranted for auditability, provenance, and accountability of AI-produced analyses (e.g., ensuring citations, reproducibility artifacts, and clear responsibility).
• Research & investment priorities:
  • Invest in domain-specific model evaluation, tooling for automated verification (e.g., citation-checkers, test-suite generators), and training programs that shift worker skills toward oversight and interpretation.
  • Cost–benefit analyses of deploying AI for data science should include verification and risk mitigation overheads, not just direct task automation gains.

If you want, I can (a) extract the rubric items and their point allocations into a compact table, (b) draft sample policy recommendations for a firm considering deploying these models for data analysis, or (c) produce a short slide-ready summary with the key charts (overall scores and category breakdown). Which would be most useful?