Frontier models now differ more in consistency than in peak ability: repeated deterministic tests reveal whether a failure is a reproducible miss fixable by an operating rule or a scattered error that requires changing the model; the proposed harness is cheap, transparent and directly decision-guiding.

Main Finding

Precision (output consistency, i.e., low variance or tight grouping across repeated identical requests), not capability (mean or best-case accuracy), is the practically decisive frontier metric for deployed AI systems. Measuring precision on deterministic, verifiable tasks (repeat many independent runs and record pass/fail) cheaply identifies whether failures are zeroable by operating discipline (skills/rules) or require changing the model/sampling. Benchmarks that report central tendency miss this axis, so procurement, engineering investment, and evaluation should prioritise measured precision and the paired “raw model vs. skill-library” delta.

Key Points

• Two independent axes: accuracy (where the mean shot lands) and precision (how tight the group is). Frontier models have converged on accuracy; they differ on precision.
• Operational consequence (marksman analogy):
  • Tight-group off-centre errors are correctable by sight-adjustment (skills/rules).
  • Scattered errors are not correctable by rules; require different model or sampling.
• Measurement proposal:
  • Use deterministic, binary-scored tasks (compile/tests/typecheck/linter) so the scorer is outside the model loop.
  • For each task run N independent samples (fresh agents) at fixed, recorded sampling config; record binary outcomes.
  • Metrics: mean pass rate mean(p_i) (accuracy), grouping score = fraction of tasks with p_i ≈ 0 or ≈ 1 (precision), per-task variance p_i(1−p_i), worst-case min p_i and all-pass Π p_i.
  • Decision rule: use pass rate to screen; inspect failing samples to determine homogeneity of causes. If failures share a single cause → tight group (zeroable). If heterogeneous causes → wide group (not zeroable).
• Worked example (illustrative):
  • Rust suite of 6 deterministic tasks, N=5, provider-default temperature.
  • Raw Opus 4.8: accuracy 0.833, grouping 1.0, worst-case 0.0: five tasks decisive-pass, one decisive-fail (p=0).
  • Single systematic parse bug identified; adding one rule in the skill library moved that task from 0/5 to 5/5. Replicated on a later default model (raw 1/5 → library 5/5).
• Important empirical caution: authoring tasks from your own rulebook tends to produce zero delta because capable models often already embody explicit good practice. The library’s value shows up only on residual gaps found by representative measurement, not on constructed tests.
• Practical constraints and limits:
  • Cost scales with N; keep suites small and periodic (quarterly gates or pre-adoption).
  • Deterministic scorers can bias against disciplined outputs (shape mismatch, test harness issues); scorer design matters.
  • Temperature/sampling must be recorded and controlled within run-pairs; cross-pair absolute p_i comparisons are weak unless sampling is anchored.
  • Cause-homogeneity judgement requires human (or judged) reading of failing samples — not fully deterministic.
  • Method applies to verifiable engineering tasks, not open-ended generative work where scatter may be acceptable or desirable.

Data & Methods

• Measurement harness:
  • Fix suite T of m deterministic, verifiable tasks.
  • Fix configuration C (model, optional skill library, recorded sampling temperature).
  • For each task, run N independent samples on fresh agents; record binary pass/fail outcomes.
• Metrics computed per suite and per-task:
  • Accuracy: mean(p_i).
  • Grouping/precision: fraction of tasks with p_i ≥ 1−ε or p_i ≤ ε (e.g., ≥9/10 or ≤1/10).
  • Scatter measure: per-task variance p_i(1−p_i) (max at p_i = 0.5).
  • Worst-case reliability: min_i p_i and all-pass Π_i p_i.
• Decision procedure:
  • Screen by pass rates.
  • Read failing samples to classify failures as homogeneous (zeroable) or heterogeneous (not zeroable).
  • Action: write/version a skill and re-measure if zeroable; change model or lower temperature if scatter.
• Empirical run in paper:
  • Suite: six Rust verification tasks.
  • N = 5 runs per task.
  • Provider-default temperature recorded but not independently controlled.
  • Observed raw→library deltas and replication across a later default model; found and fixed a systematic parse bug with a single rule.

Implications for AI Economics

• Product differentiation and procurement:
  • Precision (consistency) becomes a scarce dimension of product value distinct from headline capability metrics. Buyers should demand measured precision (grouping scores, worst-case pass rates) in SLAs and procurement decisions.
  • Leaderboards that report mean or peak capability will understate production value; vendors may compete on precision rather than raw capability.
• Cost-benefit of skill libraries vs model upgrades:
  • The operating-discipline (skills/rules) has measurable value only on residual, model-specific gaps. As base models improve, marginal returns to large rulebooks decline toward zero.
  • Investment strategy: measure where failures are homogeneous and invest in small, live, measurement-fed rule libraries; for heterogeneous failures, budget for model upgrades or sampling/temperature changes.
• Total cost of ownership and reliability pricing:
  • Precision-driven evaluation exposes tail risks and reliability costs (rework, human oversight, downtime). Firms should price models not only by cost-per-token or average accuracy but by expected operational failure variance and mitigation costs.
  • Insurance, SLAs, and regulatory compliance can be structured around precision metrics (e.g., minimum grouping scores, guaranteed all-pass probabilities on a representative deterministic suite).
• Operational labor and automation:
  • Where failures are predominantly tight and zeroable, investment in operating discipline allows more effective automation and compounding improvement (rules retire classes of failures). Where failures are scattered, automation risk remains high and human oversight costs persist.
• Market incentives and benchmarking:
  • Benchmarks should add repeat-sampling deterministic tasks and report grouping/worst-case metrics to align market incentives with production value.
  • Vendors may be incentivised either to reduce variance (improve precision) or to expose tuning/sampling knobs to buyers; contracting should recognise which lever is needed.
• Measurement & governance investments:
  • Firms should budget for periodic precision reviews (N× evaluation cost, small suite) and for tooling to ensure independent runs, deterministic scorers, and contextual metadata (run IDs, arm identifiers).
  • Regulators and standards bodies can require precision measurement for certification in high-stakes domains; precision-based certification is feasible (deterministic tasks + grouping score) and directly tied to operational tractability.
• Macro implication:
  • As frontier capability saturates, the economic competition frontier shifts to precision and operating-discipline effectiveness. This will influence where R&D and product investment flows (improving sampling consistency, model calibration, measurement infrastructure, or targeted rule libraries).

Concluding practical recommendation: procurement and deployment pipelines should add a small deterministic-sample precision gate (fixed suite, N runs, paired raw vs library measurement, inspect failure causes). Use grouping and worst-case metrics to decide whether to invest in skills (if failures are homogeneous) or in model/sampling changes (if failures are heterogeneous).