0 cumulative citations
View corpus contextTurning requirements into executable specifications, the CAPE protocol slashes model constraint violations by 81% versus standard preference-tuning and cuts annotation costs 5–20× by making specifications reusable; verification accuracy rises strongly with model scale while preference agreement plateaus.
CAPE: Capability Achievement via Policy Execution
CAPE, a Specify→Verify→Correct→Train protocol for Capability Engineering, reduces model policy-violation rates by 81% relative to DPO across 109,500 examples, improves verification accuracy with model scale (r=0.94), and lowers annotation costs and timelines by producing reusable executable specifications.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Modern AI systems lack a way to express and enforce requirements. Pre-training produces intelligence, and post-training optimizes preferences, but neither guarantees that models reliably satisfy explicit, context-dependent constraints. This missing abstraction explains why highly intelligent models routinely fail in deployment despite strong benchmark performance. We introduce Capability Engineering, the systematic practice of converting requirements into executable specifications and training models to satisfy them by default. We operationalize this practice through CAPE (Capability Achievement via Policy Execution), a protocol implementing a Specify -> Verify -> Correct -> Train loop. CAPE is grounded in two empirical findings: (1) contextual objectivity, where properties appearing subjective become objective once context is fixed (inter-annotator agreement rises from kappa = 0.42 to kappa = 0.98), and (2) verification-fidelity scaling, where verification accuracy improves with model scale (r = 0.94), unlike preference agreement which plateaus at 30 to 50 percent disagreement regardless of compute. Across 109,500 examples in six domains, CAPE reduces violation rates by 81 percent relative to DPO (standard deviation less than 0.3 percent). By replacing per-example annotation with reusable specifications, CAPE reduces costs by 5 to 20 times and shortens timelines from months to weeks. We release the CAPE protocol, PredicateGraph schema, CPL specification language, and policy packs under Apache 2.0. We also launch CapabilityBench, a public registry of model evaluations against community-contributed policies, shifting evaluation from intelligence benchmarks toward capability measurement.
Summary
Main Finding
CAPE (Capability Achievement via Policy Execution) presents a practical, scalable protocol for turning deployment requirements into executable specifications and training models to satisfy them by default. Across 109,500 evaluated examples in six domains, CAPE reduced policy-violation rates by 81% relative to DPO (σ < 0.3% for symbolic policies). Two empirical regularities underlie CAPE’s feasibility: (1) contextual objectivity — many seemingly subjective requirements become effectively objective once context is fixed (inter-annotator Fleiss’ κ rising from 0.42 → 0.98), and (2) verification-fidelity scaling — verification accuracy improves with model scale (reported correlation r ≈ 0.94), whereas preference agreement plateaus (30–50% disagreement). CAPE and its artifacts (PredicateGraph, CPL, policy packs, CapabilityBench) are released under Apache‑2.0.
Key Points
- Protocol and loop: CAPE operationalizes Capability Engineering via a Specify → Verify → Correct → Train loop. Policies are written once and reused rather than creating per-example reference answers.
- Verification spectrum:
- Structural properties: deterministic symbolic policies (CPL, PredicateGraph).
- Semantic properties: learned verifiers trained on explicit rubrics with meta‑verification.
- Only genuinely subjective properties (e.g., aesthetic quality, humor) require preference learning.
- Empirical findings:
- Contextual objectivity: annotation study (500 outputs, 5 annotators, 3 conditions) — Abstract κ = 0.42, Contextualized κ = 0.73, Explicit Policy κ = 0.98.
- Verification-fidelity scaling: verification accuracy correlates strongly with model scale (r ≈ 0.94); verification improves where preference signals do not.
- Large evaluation: 109,500 examples, six domains; CAPE reduces violations by 81% vs DPO; 96.2% compliance achieved without inference-time guardrails in experiments.
- Why CAPE is now practical:
- Long context windows enable full-output extraction.
- Instruction following and structured generation (JSON/grammar modes, constrained decoding) have improved extraction fidelity above a necessary threshold (~90%).
- Comparisons:
- Sidesteps RLHF/DPO pathologies (preference ceiling, reward-shaping biases like length bias).
- More deterministic and auditable than Constitutional-AI style natural-language principles.
- Complementary to formal methods and RLVR but emphasizes reusable, versioned policies rather than per-example ground truth.
- Infrastructure released: CPL specification language, PredicateGraph schema, policy packs, trained verifier tooling, and CapabilityBench (public registry for policy-based evaluations).
Data & Methods
- Inter-annotator study:
- N = 500 model outputs; 5 human annotators.
- Three conditions per output: Abstract judgment, Contextualized judgment, Explicit policy (CPL) run.
- Metrics: Fleiss’ κ and percent agreement (Abstract κ = 0.42, 63.2% agreement; Contextualized κ = 0.73, 84.7%; Explicit Policy κ = 0.98, 98.9%).
- Large-scale evaluation:
- 109,500 examples across six domains (medical, code, finance, workflows, extraction, reasoning — per paper).
- Baselines: DPO and other preference-based post-training approaches.
- Outcomes: 81% reduction in policy violations vs DPO; stability σ < 0.3% for symbolic policies.
- Verification experiments:
- Measured verification accuracy as a function of model scale; reported correlation r ≈ 0.94 between scale and verifier fidelity.
- Compared preference disagreement (30–50% persistent) vs verification improvements with scale.
- Tools and constructs:
- PredicateGraph: structured representation for outputs amenable to symbolic checks.
- CPL: policy/specification language for expressing verifiable constraints.
- Learned verifiers: trained on explicit rubrics, with meta‑verification layers to catch hallucinated verifier outputs.
- External/related validation:
- Cites concurrent/independent results (e.g., Su et al., DeepSeek/AlphaProof lines) showing high inter-model agreement when explicit references/rubrics are provided and that compact verifiers can generalize across domains.
Implications for AI Economics
- Cost and time to deploy reliable capabilities fall substantially:
- Reusable specifications replace per-example annotation, yielding reported cost reductions of 5–20× and shortening timelines from months to weeks. This alters the unit economics of post-training and deployment.
- Changed labor composition and markets:
- Demand shifts from large-scale preference annotation toward policy engineering, verifier training, and specification design. New specialist roles (policy pack authors, verifier engineers, capability auditors) will command value.
- Markets may emerge for third-party policy packs and certified verifier models; firms can productize domain-specific capability bundles.
- Productization and monetization:
- Capability guarantees (percent compliance against published policies) become a sellable attribute. Buyers may pay premiums for models/services with verifiable, auditable compliance SLAs.
- Pricing models could move from compute/throughput metrics to capability‑guarantee tiers and policy support services.
- Competitive dynamics and lock-in:
- Firms that accumulate high-quality, reusable policy libraries and validated verifiers can obtain durable advantages (specification networks, regulatory certifications, client-specific policy histories).
- Open registries like CapabilityBench may commoditize policy evaluation standards, reducing asymmetric information and enabling more transparent procurement.
- Regulation, liability, and auditability:
- Executable specifications and traceable verification produce audit trails valuable for compliance and liability management (FDA/HIPAA, financial regs). Regulators may require capability-evidence rather than aggregate benchmark scores.
- Insurers and legal frameworks can price risk more accurately when violations are measurable and remediable.
- Macro and adoption considerations:
- CAPE increases the marginal value of model scale insofar as verification fidelity scales with size; firms will weigh model scale investments against gains in verifiable capability (not just raw benchmark performance).
- For many enterprise applications, the bottleneck shifts from model intelligence to specification bandwidth (time/skill to write good policies).
- Research and policy priorities for economists:
- Estimate the value of reusable policy assets and its effect on firm valuation and market structure.
- Model labor reallocation effects (annotators → policy engineers, verifier builders).
- Study procurement dynamics: how CapabilityBench-like registries affect price competition and market entry.
- Quantify welfare and regulatory impacts when capability guarantees reduce externalities (e.g., medical errors, financial misadvice).
- Caveats and open questions:
- External validity: CAPE’s empirical claims derive from the reported datasets/domains; generalization to all tasks depends on whether properties are contextually objectifiable and whether verifier fidelity continues to scale.
- Some capabilities remain inherently subjective and still require preference learning.
- Learned verifiers are probabilistic; miscalibration or adversarial inputs remain risks.
- Strategic behavior: vendors might overfit policies to benchmarks or game registries; governance of policy packs and registries matters.
Suggested next steps for AI-economics researchers: - Empirically estimate cost savings and turnaround improvements in representative enterprise settings (healthcare, finance, legal). - Model how reusable policy assets change competition and returns-to-scale in ML service markets. - Explore regulatory and liability frameworks that leverage executable specifications and public capability registries.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports large-sample results (109,500 examples), strong effect sizes (81% reduction in violations), and scaling correlations (r=0.94), and it releases artifacts (spec languages, policy packs). However, evidence is limited by non-randomized comparisons, potential selection and measurement choices (domains, specification quality, baseline implementation), limited detail here on statistical controls or robustness checks, and little external validation on independently collected datasets or deployment outcomes.
Methods Rigormedium — Rigor is supported by large sample size, multiple domains, quantitative metrics (kappa, correlation, violation rates), and publication of tooling and datasets, which aids reproducibility. But the summary lacks detail on dataset construction and representativeness, baseline selection and tuning, pre-registration or held-out validation procedures, the exact verification protocols, and robustness analyses (e.g., adversarial examples, cross-cultural annotation). Those gaps leave open alternative explanations for the magnitude of reported gains.
Sample109,500 examples across six domains (domains not enumerated here); evaluations use models at multiple scales to measure verification-fidelity scaling (r = 0.94); human annotation shows inter-annotator agreement rises from kappa = 0.42 to 0.98 when context/specifications are fixed; comparisons are made against a standard DPO baseline; authors release CAPE protocol, PredicateGraph schema, CPL spec language, and policy packs under Apache 2.0.
Themesgovernance adoption org_design
IdentificationComparative evaluation: the authors implement the CAPE protocol and compare model violation rates, verification accuracy, and annotation costs against a baseline (DPO) across datasets and model scales; they also report correlations (e.g., scale vs verification accuracy). No randomized assignment or natural experiment is reported, and causal claims rest on before/after and cross-model comparisons rather than an identification strategy designed to rule out confounders.
GeneralizabilityResults may depend on the six chosen domains; performance may differ on other tasks or open-ended generation., Depends on availability and quality of executable specifications — requires expertise that may not exist in all organizations., Reported scaling relationships may hold only for the model families and scales tested; different architectures or smaller models may behave differently., Baselines (DPO) and implementation details affect comparability — improvements may shrink with stronger or differently tuned baselines., Cultural and linguistic contexts not specified; specifications that work in one context may not transfer., Adversarial or long-tail safety issues and real-world deployment dynamics (user interactions, evolving distributions) may reduce effectiveness.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Modern AI systems lack a way to express and enforce requirements; pre-training produces intelligence and post-training optimizes preferences, but neither guarantees that models reliably satisfy explicit, context-dependent constraints. Ai Safety And Ethics | negative | ability to express and enforce requirements / reliable satisfaction of context-dependent constraints |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| We introduce Capability Engineering, the systematic practice of converting requirements into executable specifications and training models to satisfy them by default, operationalized through CAPE (Specify -> Verify -> Correct -> Train). Ai Safety And Ethics | positive | method for ensuring models satisfy requirements (operational protocol) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| Contextual objectivity: properties appearing subjective become objective once context is fixed (inter-annotator agreement rises from kappa = 0.42 to kappa = 0.98). Decision Quality | positive | inter-annotator agreement (kappa) |
Reading fidelity
high
Study strength
medium
|
inter-annotator agreement rises from kappa = 0.42 to kappa = 0.98
|
| Verification-fidelity scaling: verification accuracy improves with model scale (r = 0.94). Decision Quality | positive | verification accuracy |
Reading fidelity
high
Study strength
medium
|
r = 0.94
|
| Preference agreement plateaus at 30 to 50 percent disagreement regardless of compute (i.e., preference agreement does not scale and retains 30–50% disagreement). Decision Quality | negative | preference agreement / disagreement rate |
Reading fidelity
high
Study strength
medium
|
30 to 50 percent disagreement
|
| Across 109,500 examples in six domains, CAPE reduces violation rates by 81 percent relative to DPO (standard deviation less than 0.3 percent). Error Rate | positive | violation rate (rate of failing to satisfy specified requirements) |
Reading fidelity
high
Study strength
medium
|
n=109500
81 percent reduction
|
| By replacing per-example annotation with reusable specifications, CAPE reduces costs by 5 to 20 times and shortens timelines from months to weeks. Training Effectiveness | positive | annotation/training cost and development timeline |
Reading fidelity
high
Study strength
medium
|
5 to 20 times (cost reduction); timeline shortened from months to weeks
|
| We release the CAPE protocol, PredicateGraph schema, CPL specification language, and policy packs under Apache 2.0. Other | positive | availability of code/specs under permissive license |
Reading fidelity
high
Study strength
high
|
not reported
|
| We launch CapabilityBench, a public registry of model evaluations against community-contributed policies, shifting evaluation from intelligence benchmarks toward capability measurement. Research Productivity | positive | availability and stated mission of CapabilityBench (evaluation registry) |
Reading fidelity
high
Study strength
low
|
not reported
|