Organizations should evaluate humans and AI on shared capability 'profiles' rather than disparate metrics; doing so makes task allocation more predictive, explainable and auditable, but requires new measurement standards and infrastructure.

Main Finding

Profile-driven suitability evaluations — inferring latent constructs (capabilities, propensities) from observed performance and placing humans and AI systems on shared scales — are the most promising approach for task-allocation decisions as AI is integrated into workplaces. This approach enables comparisons that are predictive of novel-task performance, explanatory about why agents succeed or fail, and auditable, addressing scalability and comparability problems inherent in credential- or benchmark-driven evaluations.

Key Points

• Reverse Turing Test dynamic: allocation algorithms increasingly judge whether a human or machine should perform a task, reversing traditional evaluation roles and creating adversarial/competitive interactions between human and AI workers.
• Current evaluation mismatch:
  • Human assessments rely on credentials, references, and interviews that do not scale or translate to AI systems.
  • AI assessments rely on benchmarks, leaderboards, and red teaming that are not directly comparable to human measures.
• Proposal: adopt profile-driven evaluations that infer latent traits (e.g., domain capabilities, reliability, bias propensity, error modes, adaptivity) from observed performance across tasks and contexts.
• Desirable properties of profile-driven evaluation:
  • Shared scales across humans and machines for direct comparison.
  • Predictive validity for performance on novel or out-of-distribution tasks.
  • Explainability: profiles illuminate mechanisms and failure modes.
  • Auditability and traceability: tests and inferred constructs can be inspected, certified, and monitored over time.
• Practical considerations:
  • Profiles should combine multi-task performance, stress tests, and behavior under adversarial or strategic conditions.
  • Measurement design must emphasize reliability, validity, and invariance across populations and system versions.
  • Profiles must be continuously updated to reflect learning, model updates, and human skill drift.
• Tradeoffs and risks:
  • Gaming and strategic behavior by agents (or vendors) requires robust, dynamic testing and anti-gaming design.
  • Building shared metrics entails institutional coordination, standardization, and possibly regulation.
  • Equity and fairness concerns: profiles must account for distributional impacts and avoid embedding biased training signals.

Data & Methods

• Conceptual approach: measurement theory / psychometrics extended to include AI systems.
• Core statistical tools:
  • Item Response Theory (IRT) and latent-trait models to infer abilities/capacities from task-response patterns.
  • Factor analysis and structural equation models to identify and validate latent constructs (capability dimensions, propensities).
  • Bayesian hierarchical models to pool information across agents, tasks, and firms while quantifying uncertainty.
  • Measurement invariance testing to ensure scales are comparable across humans and AI systems, and across demographic or system subgroups.
  • Causal inference and transportability methods to estimate generalization to novel tasks and environments.
  • Experimental designs (A/B tests, multi-armed bandits) to evaluate allocation policies and feedback loops in live settings.
  • Stress-testing and adversarial evaluation to reveal failure modes and propensities under strategic conditions.
• Data requirements:
  • Diverse, representative task sets sampled to probe multiple capability dimensions and contexts.
  • Longitudinal performance logs to capture adaptation, learning, and drift.
  • Metadata on task characteristics, cost, risk, and contextual constraints to enable conditional predictions.
  • Auditable records for provenance, versioning, and reproducibility.
• Validation criteria:
  • Predictive validity: correlation of profile-inferred traits with out-of-sample or novel-task performance.
  • Construct validity: coherence with theory and convergent evidence from multiple tests.
  • Reliability: stability of trait estimates under repeated measurement.
  • Robustness: resistance to gaming, manipulation, and distribution shifts.

Implications for AI Economics

• Labor allocation and substitution/complementarity:
  • Shared-profile metrics reduce frictions in matching tasks to agents, enabling more efficient allocation between human and machine labor and clearer assessment of substitution vs complementarity across task types.
  • Firms can better identify where AI augments human labor (complementary capabilities) versus displaces it (where AI profiles outperform humans).
• Productivity measurement and accounting:
  • Profiles provide means to measure and compare “AI capital” to human capital on common axes, improving productivity accounting, valuation of AI investments, and ROI analyses.
• Hiring, contracting, and compensation:
  • Performance- and profile-based contracting could replace credentials- or title-based hiring, shifting compensation toward demonstrable latent capabilities and propensities.
  • Reduces search and information costs for firms but introduces new costs for standardized testing infrastructure and ongoing monitoring.
• Market structure and competition:
  • Standardized, auditable profiles could lower entry frictions for new AI providers by enabling transparent comparison, but they could also lead to consolidation if large firms control testing infrastructures or datasets.
• Regulation and governance:
  • Regulators may need to set validation standards, disclosure requirements, and auditing processes for profile-driven evaluations to ensure fairness, safety, and accountability.
  • Profile-based audits can help enforce sector-specific constraints (e.g., high-reliability requirements in healthcare or finance).
• Dynamics and strategic behavior:
  • Vendors and individuals will have incentives to optimize for profile metrics; the measurement system must be designed to reduce perverse incentives and to detect overfitting to benchmarks.
  • Continuous updating of profiles is necessary as models learn and human skills evolve, affecting labor demand trajectories and retraining needs.
• Equity and distributional effects:
  • Profile-driven allocation can improve fairness by using task-relevant performance measures, but if underlying data or tests reflect societal biases, profiles may reproduce or amplify inequalities. Policy and design choices will matter for distributional outcomes.
• Research and data infrastructure:
  • Economic research will require new datasets linking agent profiles to firm outcomes, wages, task allocation decisions, and macro-level productivity to quantify the broader impacts of profile-driven allocation.

Overall, adopting profile-driven suitability evaluations reshapes how firms, workers, and policymakers measure and allocate labor in the age of AI. It promises greater predictive power and comparability across human and machine agents but requires careful measurement design, governance, and infrastructure to manage incentives, fairness, and dynamic effects.