Automatically generated LLM ‘skill’ files do not boost data‑science task performance: a large, multi‑model experiment finds no significant improvement from full or ablated generated skills versus task-only prompts.

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

Wei-Jung Huang · July 08, 2026

arxiv quasi_experimental medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Across 56 data-science tasks, nine model configurations, and three providers, LLM-generated reusable skills did not significantly improve single-shot task performance compared with task-only prompting, and ablations and a token-matched control show no reliable benefit.

Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.

Summary

Main Finding

LLM-generated, low-curation “skill” files (one per data‑science workflow stage) do not improve automated data‑science task performance compared with task-only prompts. Across 56 verifiable tasks, nine model configurations, and 10,584 total runs, neither full generated skills nor any ablated component produced a statistically reliable gain; the largest spread across conditions was only 1.2 percentage points and all reported p-values were ≥ 0.396.

Key Points

Scope: Four lifecycle stages—data preparation, data extraction (SQL), statistical analysis, and reporting—mapped to 56 tasks (14 per stage).
Skill format: One LLM-generated skill per stage (produced by Gemini 2.5 Pro without human editing). Each skill had four sections: Routing, Core Procedure, Worked Examples, Reference Notes.
Ablation conditions: No-Skill (task-only), Core-Only, Core+Examples, Core+Refs, Full. Supplemental controls: token-matched Length-Control (task-irrelevant office-supply skill) and Full+Priority (directive saying task overrides skill).
Scale: 56 tasks × 5 main conditions × 9 models × 3 repetitions = 7,560 main runs; plus 1,512 runs each for Length-Control and Full+Priority → 10,584 total runs. 93.4% of main-ablation cells were unanimous across repetitions (majority-vote used).
Models: Nine configurations from three providers (OpenAI, Google, Anthropic), including compact, frontier, and explicit-reasoning settings.
Verification: Deterministic, automated pass/fail verifiers per stage (CSV/DataFrame equality, SQL execution, numeric tolerance for stats, JSON schema validation).
Aggregate results: No-Skill 68.3% pass rate vs Full 67.5% (aggregate spread 1.2 pp). Mixed-effects model coefficients for skill conditions were near zero and non-significant (e.g., Full β̂ = −0.008, p = 0.497). McNemar tests show helps vs hurts roughly balanced.
Token/control check: Length-Control (token-matched, but task-irrelevant) scored 66.9% vs Full 67.5% and No-Skill 68.3% — no evidence that data‑science skill content beats matched-length irrelevant content.
Costs: Full skills substantially increased input tokens (mean ~1,293 tokens vs ~287 for No-Skill), adding cost without measurable accuracy gains.
Task-source robustness: Null result holds both for self-authored and externally grounded tasks (no meaningful difference).

Data & Methods

Skill generation: Single, autonomous generation session per lifecycle stage using Gemini 2.5 Pro (no human selection/refinement). Ablations produced by mechanically deleting sections.
Tasks: 56 verifiable tasks—half self-authored (drafted by an agent) and half externally grounded (public datasets/APIs/benchmarks). Gold outputs fixed and validated programmatically before skill generation.
Models & runs: Nine model configurations × 56 tasks × 5 main conditions × 3 repetitions (plus supplemental conditions) → 10,584 total runs.
Outcome measurement: Fully automated binary pass/fail checks per task type; programmatic inspection of condition-flip cases confirmed substantive errors (not just formatting).
Analysis: Linear mixed-effects model (condition fixed effect; task and model random effects), group-stratified bootstrap, bootstrap CIs (10k iterations), and McNemar paired tests. Results consistent across methods.

Implications for AI Economics

Low-curation automation may not yield expected productivity gains. Generating a single static skill per task family (no human curation) appears to be a low-return investment for the verifiable execution layer of data‑science work—accuracy gains are absent while token/computational costs rise.
Cost vs benefit: Organizations should treat automatically generated skills as an explicit cost center (longer prompts, more tokens) and measure downstream ROI before wide deployment. Token and API costs accumulate across many tasks and users.
Labor allocation: The null results suggest value in focusing scarce human engineering effort on higher‑impact interventions (expert-curated skills, interactive orchestration, validators, tool integration) rather than on fully automated single-shot skill generation.
Product design & tooling: Static, flat skill injection is a weak strategy for improving one-shot task accuracy. Alternatives likely to be more productive economically include:
- Selective loading / retrieval-augmented skill access (load only relevant guidance),
- Structured tool/schema integrations and multi-turn orchestration (validation and feedback loops),
- Human-in-the-loop curation or light vetting of generated skills,
- Task-level priority mechanisms or system-level instruction hierarchies (since flat skill injection can create conflicting advice).
Risk management: Relying on uncurated generated skills at scale risks adding cost and occasional conflicts/harm without net accuracy benefit. Firms should monitor not just aggregate pass rates but task-level flips (helps vs hurts) and consider guardrails (priority directives, unit tests) when experimenting with skill injection.
Research & purchasing decisions: Benchmarks and procurement that promise productivity gains from “skill libraries” should require evidence for curated or orchestrated implementations, not just LLM-generated, flat-injected skill files.

Limitations and caveats - The study tests a specific low-curation workflow (single-shot, flat injection of LLM-generated skills produced by one generator). It does not evaluate expert-curated skills, iterative refinement of generated skills, or selective/conditional loading architectures that could change outcomes. - Results pertain to execution/verifiable tasks; higher-level judgment work (product framing, experiment design) was intentionally excluded and may react differently to skill-style guidance. - Findings are robust across multiple models/providers in this study but generalization to future model releases or different generation protocols requires further testing.

Bottom line: For verifiable, execution‑level data‑science tasks, one-off LLM-generated skill files (flat-injected, no human curation) do not improve accuracy and add token cost—organizations should not adopt this as a default single‑shot prompting strategy without further curation, orchestration, or validation.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — Large-scale, pre-specified comparisons with many runs (7,560 plus 1,512 control runs) and multiple providers give strong internal evidence that generated skills do not improve single-shot task performance in the tested setup; however, the study does not measure real-world productivity, user outcomes, or multi-turn workflows, limiting external validity and economic interpretation. Methods Rigorhigh — The paper uses a systematic, well-powered design with extensive ablations, multiple model/provider configurations, a token-matched control, and statistical tests; weaknesses are largely about external validity (synthetic/benchmarked tasks, single-shot prompting, lack of human-in-the-loop evaluation) rather than internal methodological flaws. Sample56 data-science tasks covering four lifecycle stages (data preparation, data extraction, statistical analysis, reporting); experiments run across nine model configurations and three providers, totaling 7,560 primary runs and 1,512 supplemental token-matched control runs; performance evaluated by task-specific metrics under No-Skill, Full skill, and multiple ablated skill variants. Themesproductivity human_ai_collab IdentificationControlled head-to-head comparisons and ablation experiments: performance on 56 data-science tasks was measured under (a) No-Skill prompting, (b) Full generated skills, and (c) multiple ablated/variant skill conditions across nine model configurations and three providers; statistical tests (p-values) and a token-matched control were used to assess differences. GeneralizabilityFindings apply to single-shot, one-skill-per-stage prompting and may not generalize to multi-turn or interactive workflows., Tasks appear benchmark-like; results may differ on real-world, messy projects or domain-specific data., Only certain LLM providers and configurations were tested; other models or tool-enabled agents might behave differently., Does not measure downstream productivity, time saved, or user satisfaction in human-in-the-loop settings., Study uses one generated skill per lifecycle stage—results may not extend to curated or hybrid (human+LLM) skill maintenance strategies.

Claims (7)

Claim	Direction	Outcome	Confidence & Evidence	Details
We find no reliable improvement from full generated skills over No-Skill prompting. Output Quality	null_result	task performance (accuracy / success rate)	Reading fidelity high Study strength high	n=7560 0.8
The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Other	null_result	number of experimental runs	Reading fidelity high Study strength high	n=7560 0.8
Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396. Output Quality	null_result	task performance (accuracy / success rate)	Reading fidelity high Study strength high	n=7560 p >= 0.396 0.8
The total spread across variants is only 1.2 percentage points. Output Quality	null_result	task performance (difference between best and worst variant)	Reading fidelity high Study strength medium	n=7560 1.2 pp 0.48
A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. Output Quality	null_result	task performance (Full skills vs token-matched irrelevant content)	Reading fidelity high Study strength high	n=1512 0.8
We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. Other	null_result	coverage of lifecycle stages tested	Reading fidelity high Study strength medium	n=4 0.48
The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy. Other	negative	suitability of single-shot generated skill as default prompting strategy	Reading fidelity high Study strength medium	n=7560 0.48