A capability-based profiling pipeline helps firms scope AI deployment by matching AI strengths to task demands, showing systems differ more across cognitive dimensions than by model family; combined with task-weighting from 410 employees, the tool flags promising pilots and areas where current models are likely ill-suited.

Main Finding

The authors introduce a practical, capability-based pipeline to assess whether an AI system is suited to specific workplace tasks. By profiling agents on a shared set of core cognitive capabilities (derived from annotated benchmark items) and eliciting the relative importance of those same capabilities for workplace activities (from domain experts), the method produces commensurable suitability scores that help organisations scope where AI deployment is promising and where it is likely to fail. They validate capability recovery on synthetic agents, profile six AI systems, and collect task-requirements weights from 410 employees across six occupational domains. Key empirical findings: AI systems differ more along cognitive dimensions than by model family, while workplace activities converge around a shared cognitive core.

Key Points

• Problem addressed: Aggregate benchmarks and expert forecasts are poor guides for real-world deployment because AI systems have jagged, non-human-like capability profiles and real tasks combine demands in novel ways.
• Pipeline overview:
• Cognitive capability profiling — infer an agent’s capability levels from performance on a demand-annotated benchmark battery.
• Task requirements weighting — elicit from domain experts the relative importance (weights) of the same cognitive capabilities for target workplace tasks.
• Suitability mapping — combine capability profiles and requirement weights to estimate relative task suitability.
• Capability taxonomy: 18 core cognitive capabilities grouped into four families: memory systems, executive control, object & space understanding, and social & communicative capabilities.
• Rubric-based annotation: Each benchmark item is annotated for per-capability demand on a six-point scale (0–5). Annotation follows the ADeLe-style rubric approach and uses LLM judges as scalable proxies to apply human-crafted rubrics.
• Modelling detail: Demand level dj,k for item j and capability k is converted to raw difficulty δj,k = e^{λ dj,k} (authors use λ = 1 in experiments), giving a geometrically increasing difficulty scale across levels.
• Validation & data:
  • Capability recovery tested on synthetic agents to show the inference procedure can recover known capability profiles.
  • Six AI systems profiled (paper does not rely on aggregate single-score comparisons).
  • Task requirements weights elicited from 410 employees across six occupational domains.
• Practical framing: The approach produces a comparative scoping tool (identifies promising pilot targets and likely failure modes) but does not replace piloting or threshold-based validation. Importance weights are relative and indicate what matters most for a task rather than whether an agent clears an absolute demand threshold.
• Mitigations mentioned: Circularity concern (using LLMs to annotate items used to evaluate LLMs) is mitigated by treating the annotator as a rubric applier and by anchoring rubrics in human expert judgment.
• Extensibility: Framework can in principle profile humans on the same capability space (by sampling annotated items as psychometric tasks) and extend to human+AI pairings to inform task allocation.

Data & Methods

• Capability set and rubrics:
  • 18 capabilities selected from psychometrics and cognitive science literature.
  • Per-capability 6-level rubrics (0–5) with level descriptors and examples to standardise annotations.
• Benchmark annotation:
  • Existing benchmarks annotated at item-level across the 18 capabilities using rubric-applying methods (LLM judges used to scale annotations).
  • Resulting annotated catalogue becomes the profiling battery.
• Capability inference:
  • Agents’ performance across annotated items is used to infer capability levels; demand levels are mapped to exponential difficulty (δ = e^{λ d}).
  • Inference model recovers latent capability parameters from observed successes/failures across items with differing demand profiles.
• Validation:
  • Synthetic agents created with known capability profiles used to test recovery accuracy.
  • Six real AI systems profiled by running them on the annotated battery.
• Task requirements elicitation:
  • Domain experts (410 employees spanning six occupational domains) provided relative importance weights for the 18 capabilities per activity/task.
  • Weights produce a requirement vector per task that is commensurable with agent capability vectors.
• Suitability mapping:
  • Capabilities weighted by task importance produce a scalar or ranked suitability measure indicating relative fit.
• Reproducibility: Code, rubrics, and benchmark annotations are made available in the project repository.

Implications for AI Economics

• Better scoping reduces uncertainty and deployment risk, addressing a top barrier to adoption (reliability). More predictable failure modes enable targeted guardrails and human-in-the-loop designs that preserve efficiency gains.
• Investment & adoption decisions:
  • Organisations can prioritise piloting and procurement based on relative suitability instead of coarse benchmarks, improving ROI of AI pilots and reducing wasted trials.
  • Capability profiling supports dynamic procurement: buy or build models that match high-weight capabilities for core tasks (market segmentation by capability profile rather than only by model family).
• Labour market effects:
  • Finer-grained task-level suitability estimates can improve predictions about which tasks/occupations are automatable, augmentable, or best left to humans — refining existing exposure and displacement models.
  • Understanding jagged capability profiles highlights opportunities for task reallocation and role redesign (complementarity), rather than binary automation predictions; this can affect wage dynamics and skill premium calculations.
• Human capital and training:
  • Profiling can inform which human skills will remain complementary (high-weight capabilities where AI is weak), guiding training and reskilling investments.
• Product and service design:
  • Firms can design human–AI workflows and handoff protocols where the mapping shows strong complementarities or predictable failure modes, enabling higher-quality services and reduced liability exposure.
• Policy and regulation:
  • Capability-based mappings give regulators a more concrete basis for assessing deployment risk by task and domain, supporting targeted regulation or sector-specific standards.
• Measurement and macro forecasts:
  • Incorporating capability-task mappings into macroeconomic models and productivity analyses can yield more accurate forecasts of AI-driven productivity gains because the approach accounts for task heterogeneity and non-uniform model capabilities.
• Limitations that affect economic conclusions:
  • The method delivers relative suitability, not hard thresholds — pilot outcomes and end-to-end deployment testing remain necessary to estimate realized productivity gains or cost savings.
  • The annotated battery and rubrics must be maintained and updated as models evolve; economic analyses should account for dynamic capability change and model retraining cycles.
  • Potential inflation on some annotated items (if they overlap with model training data) could bias capability estimates; economic decisions should therefore be complemented by deployment-stage monitoring.

Overall, this capability-based pipeline provides a principled, scalable intermediate step between high-level benchmark scores and full-scale deployment trials. For AI economics, it refines how we model adoption, complementarity, and productivity impacts by shifting focus to structured mappings between task demands and jagged model capabilities.