0 cumulative citations
View corpus contextLive trials reveal skills add measurable runtime value: ACES finds a mean composite 'Skill Lift' of ~21% across 947 paired agent trials, and shows static document scans correlate weakly with real agent performance, underscoring the need for paired, execution-based evaluation in production skill pipelines.
Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
ACES is a repository-native framework that runs paired live agent trials to measure a skill's marginal contribution (Skill Lift), finding an average composite Skill Lift of 0.2134 across 947 paired cases and demonstrating that static doc-scans poorly predict runtime impact (structural vs LLM-judge Spearman ρ = 0.14).
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman $ρ= 0.14$). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95\% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8\% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
Summary
Main Finding
ACES (Agentic Continuous Evaluation of Skills) extends static, document-based skill gating with paired live-agent evaluation of executable agent skills. Running 947 paired trials across 58 production skills and multiple harnesses, ACES reports a mean composite Skill Lift of 0.2134 (95% CI [0.1967, 0.2301]) and shows that static scans (structural checks and LLM-as-judge rubrics) are necessary but insufficient: they disagree with each other (Spearman ρ = 0.14 on a 145-skill corpus) and cannot observe runtime behaviours like discovery, routing, tool use, or workflow fidelity that materially affect task outcomes.
Key Points
- Problem motivation
- Modern LLM-agent ecosystems use reusable "skills" (SKILL.md + optional scripts) but current gates mostly scan artifacts; scanning cannot observe whether a skill actually helps an agent complete tasks at runtime.
- ACES core idea
- Treat skills as executable artifacts and evaluate them by running paired live trials: with-skill vs. baseline (same agent/model/workspace/grader; target skill toggled).
- Report Skill Lift = marginal contribution of the target skill under controlled conditions.
- Design principles
- Write once, evaluate everywhere: one evaluation contract (evals.json and optional BYOT/BYOG) that can be executed across harnesses.
- Differential measurement: paired with-skill/baseline runs isolate marginal value.
- Developer-guided evaluation: authors provide or refine datasets and expected_behaviors; LLM-assisted bootstrapping is optional but not authoritative.
- Implementation and tooling
- ATIF (Agent Trajectory Interchange Format) for portable trajectory traces across harnesses.
- Evaluator suite combines deterministic structural checks (~50 rules across Correctness / Discoverability / Reliability / Efficiency), LLM-as-judge rubric (9 base + 1 conditional criteria scored 0–10), script linting, security scanning, and runtime graders (six default runtime metrics including outcome and process signals).
- NVIDIA SkillEvaluator: an open-source implementation of ACES/Tier 3 live evaluation.
- Empirical highlights
- Static scans on 145 real skills: structural scores mean ≈79.2; 94.5% pass a permissive 70-point structural gate; LLM-judge pass rate ≈86.2%; structural vs judge scores correlate weakly (Spearman ρ = 0.14).
- Live paired evaluation: 947 scored paired cases from 58 production skills across four primary harnesses — mean composite Skill Lift = 0.2134; mean outcome-only lift (accuracy + goal-accuracy average) = 0.1799; composite lift positive in 72.8% of paired cases.
- Process metrics (skill execution, behavior checks, efficiency) often show the largest improvements and capture authoring/runtime phenomena scans cannot.
- Robustness notes
- LLM-judge absolute scores vary with judge model (spread ≈1.5 points on 0–10 between strictest/lenient judges), so judge attribution is important.
- Paired measurement reveals that as baseline agents/models improve, absolute performance can rise while marginal Skill Lift for a given skill can shrink.
Data & Methods
- Corpus and experimental scope
- 145 real skills collected from internal enterprise repositories and public catalogs for doc-scan analysis.
- Live-agent paired experiments: 947 scored paired cases, covering 58 production skills (from 64 candidates) and four primary agent harnesses.
- Static evaluation (scan-only)
- Structural checks: ~50 deterministic rules across four weighted dimensions (Correctness 0.35, Discoverability 0.25, Reliability 0.25, Efficiency 0.15). Default gate set at 70/100.
- LLM-as-judge rubric: 9 base criteria + 1 conditional (structural coherence) scored 0–10; judge models tested included Claude 3.7, Claude 3.5, and a 9B in-house model.
- Linting and security scanning applied to scripts and full content (prompt-injection, leaked secrets, destructive patterns).
- Live evaluation protocol
- Evaluation assets: an author-authored evals.json per skill (id, user question, expected_skill or negative case, expected_script optional, ground-truth answer, ordered expected_behavior list). BYOT (Bring Your Own Task) and BYOG (Bring Your Own Grader) supported for domain-specific needs.
- Paired runs: for each evaluation case, run the same agent/model/harness twice (baseline without target skill; with-skill with target skill enabled while keeping prerequisites/helper skills constant).
- Trace format: ATIF captures full interaction trajectories so the same graders run across different harnesses.
- Evaluator suite scores multiple metrics (outcome metrics such as accuracy and goal accuracy; process metrics such as whether the expected behaviors were followed, execution/routing, and efficiency). ACES aggregates these into a composite Skill Lift and reports per-dimension lifts.
- Metrics & reporting
- Skill Lift defined as the paired difference in graded metrics between with-skill and baseline conditions (reported per metric and as a composite).
- Statistical reporting includes mean lifts and confidence intervals (e.g., composite Skill Lift mean 0.2134 with 95% CI [0.1967,0.2301]).
- Limitations acknowledged by authors
- Judge-model sensitivity affects absolute LLM-judge scores.
- Evaluation quality depends on author-supplied assets; bootstrapped datasets help but are not authoritative.
- Paired lift is sensitive to baseline model/harness improvements; lift can shrink even as absolute performance increases.
Implications for AI Economics
- Measuring marginal value of modular capabilities
- Skill Lift operationalizes marginal product of a software/component in agent ecosystems — analogous to marginal productivity measures in economics — enabling evidence-based investment decisions (which skills to develop, maintain, or buy).
- Productization and marketplace design
- A standardized paired-evaluation contract (ATIF + evals.json) and public tooling (SkillEvaluator) lower information frictions in skill marketplaces: buyers can compare skills by measured lift on relevant tasks rather than relying solely on docs or marketing.
- Verified Skill Lift becomes a potential quality signal for pricing, bundling, and tiering skill/plugin offerings.
- Incentive and procurement alignment
- Firms can adopt Skill Lift in procurement and internal CI/CD to tie incentives (developer bonuses, SLAs, rollback triggers) to marginal customer-valued improvements rather than artifact-level checks.
- The developer-guided evaluation workflow supports higher-quality dataset assets, aligning author incentives with reproducible evidence.
- Risk management, model updates, and externalities
- Differential evaluation makes it practical to detect silent regressions caused by model updates (a decline in Skill Lift even when absolute baseline improves), enabling better risk controls and rollback policies.
- The methodology highlights negative externalities such as skill collisions and discovery failures — observable only at runtime — which affect aggregate productivity and coordination costs in multi-skill workspaces.
- Labor substitution and productivity accounting
- By quantifying how much a skill improves task success/efficiency, organizations can better estimate labor-savings or task automation potential attributable to particular skills, supporting more precise ROI and workforce-planning models.
- Policy and standardization
- Standardized runtime evaluation (ATIF, paired Skill Lift) supports benchmarking and regulatory auditing of agent capabilities — e.g., for claims about automation levels, safety checks, or fairness — because it observes behavior under controlled conditions rather than relying on static artifacts.
- Limitations and economic caveats
- Lift is context- and baseline-dependent: results vary by harness, model, and task distribution. Buyers and policy-makers must interpret Skill Lift conditionally (which baseline/agent/harness was used).
- Author-provided evaluation assets can embed biases or narrow task scopes; governance mechanisms (audits, third-party benchmarks) will be needed to ensure credible marketplace signals.
- Judge-model variability imposes measurement noise; standardized judge attribution and calibration will be necessary for market-grade comparisons.
Summary takeaway ACES provides a practical, portable methodology and tooling to move skill evaluation from static artifact checks to measured marginal contributions in live agent contexts. For AI economics, this enables more accurate valuation, procurement, risk management, and incentive design around modular agent capabilities — but users must account for baseline dependence, judge variability, and dataset quality when using Skill Lift as an economic signal.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper reports a substantial set of controlled, paired runtime trials (947 paired cases) and clear summary statistics (mean composite Skill Lift = 0.2134 with 95% paired-case CI), which credibly isolate the marginal value of a skill under a fixed agent/harness/grader; however, the evidence is constrained to enterprise/internal and public skill corpora, relies on author-authored evaluation assets and LLM-as-judge metrics (judge-model sensitivity reported), covers a limited set of harnesses and judge models, and lacks external replication or field-level outcome measures, limiting external validity.
Methods Rigormedium — Strengths: principled paired-difference design that holds many confounders constant, an interoperable trajectory format (ATIF), multi-tier evaluators (deterministic + LLM-judge + domain checks), and reasonably large N for paired cases. Weaknesses: evaluation assets are often authored/curated by skill authors (potential for optimistic case selection), judge scores depend on the choice of judge LLMs (reported ~1.5 point spread), unclear randomization/replication protocols and sensitivity to model/harness/versioning, and limited description of variance sources and robustness checks across broader agent/model populations.
SampleMixed-source corpus of 145 real skills drawn from internal enterprise repositories and public catalogs; static doc-scans (≈50 deterministic rules + LLM-as-judge rubrics) run on 145 skills. Live paired evaluation conducted on 947 scored paired cases drawn from 58 of 64 production skills across four primary agent harnesses. LLM-judge rotation used three judges (Claude 3.7 Sonnet, Claude 3.5 Haiku, and an in-house 9B) to assess rubric sensitivity.
Themeshuman_ai_collab productivity
IdentificationPaired within-agent experiments: for each authored task ACES runs two matched conditions (with-target-skill vs baseline-without-target) holding agent, LLM model, harness, workspace configuration, and grading policy fixed while staging prerequisite/helper skills equally; the paired difference (Skill Lift) is interpreted as the target skill's marginal causal contribution under that fixed environment.
GeneralizabilityCorpus skewed toward enterprise/internal skills and public catalogs; may not reflect consumer, academic, or other domains., Evaluation assets (evals.json, expected_behavior) are often authored or refined by skill authors, introducing potential selection or specification bias., Results depend on specific agent harnesses, LLM model versions, and judge-model choices; judge sensitivity (~1.5 points) can affect absolute scores., Paired evaluation isolates marginal effect under the tested harness/model configuration but does not directly translate to firm-level productivity, labor, or economic outcomes., Limited number of harnesses and production skills in paired trials; not a fully randomized field experiment across deployments.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| ACES evaluates skills as executable agent artifacts by running paired live trials with and without a target skill and reporting the target skill's marginal contribution as Skill Lift. Output Quality | positive | Marginal runtime contribution of a target skill to agent task performance |
Reading fidelity
high
Study strength
high
|
not reported
|
| On a corpus of 145 real skills, 94.5% passed the default structural C-grade gate, while 86.2% passed the LLM-judge rubric. Output Quality | positive | Pass rate on structural and LLM-judge skill-quality gates |
Reading fidelity
high
Study strength
high
|
n=145
94.5% structural pass rate; 86.2% LLM-judge pass rate
|
| Structural scores and LLM-judge scores showed only weak agreement across the 145-skill corpus, with Spearman correlation ρ = 0.14 and Pearson correlation r = 0.08. Output Quality | positive | Agreement between structural and LLM-judge assessments of skill quality |
Reading fidelity
high
Study strength
high
|
n=145
Spearman ρ=0.14; Pearson r=0.08
|
| Across 947 scored paired cases from 58 of 64 production skills and four primary agent harnesses, mean composite Skill Lift was 0.2134. Output Quality | positive | Composite Skill Lift across runtime evaluation metrics |
Reading fidelity
high
Study strength
high
|
n=947
mean composite Skill Lift = 0.2134
|
| The 95% paired-case confidence interval for mean composite Skill Lift was [0.1967, 0.2301]. Output Quality | positive | Mean composite Skill Lift |
Reading fidelity
high
Study strength
high
|
n=947
95% paired-case CI [0.1967, 0.2301]
|
| Mean outcome-only lift, defined as the average of accuracy and goal accuracy, was 0.1799. Output Quality | positive | Average accuracy and goal accuracy |
Reading fidelity
high
Study strength
high
|
n=947
mean outcome-only lift = 0.1799
|
| Composite Skill Lift was positive in 72.8% of paired cases. Output Quality | positive | Whether the target skill improved composite runtime performance relative to baseline |
Reading fidelity
high
Study strength
high
|
n=947
positive in 72.8% of paired cases
|
| The largest process-metric gains occurred in skill execution, behavior check, and skill efficiency. Organizational Efficiency | positive | Skill execution, expected-behavior compliance, and skill-use efficiency |
Reading fidelity
high
Study strength
medium
|
n=947
|
| The default structural gate is permissive: 94.5% of skills cleared the 70-point threshold, but only 48.9% reached 80 points. Output Quality | mixed | Distribution of structural skill-quality scores |
Reading fidelity
high
Study strength
high
|
n=145
94.5% pass the 70-point gate; 48.9% reach 80 points
|
| LLM-judge scores varied by approximately 1.5 points on a 0–10 scale across three judge models, although relative ranking within the corpus was largely preserved. Output Quality | mixed | Absolute LLM-judge scores and relative ranking of skills |
Reading fidelity
high
Study strength
medium
|
n=145
roughly 1.5 points on the 0–10 scale
|