A hybrid expert-plus-AI system sharply improves project-selection accuracy—97.5% predictive accuracy and an estimated 68% boost in portfolio value (≈$83.5m) in one firm's retrospective test—though the finding rests on a single-enterprise evaluation and needs broader validation.

Main Finding

Combining structured expert judgment (an extended Analytic Hierarchy Process, AHP) with modern machine learning in a three-step hybrid architecture produces highly accurate and economically valuable project-selection decisions. On a retrospective enterprise dataset of 10,000 software projects the hybrid system attained 97.5% accuracy, 96.9% F1-score, and 0.989 AUC-ROC (≈25% better than baselines) and is estimated to increase portfolio value by 68.2% (≈USD 83.5M incremental value) relative to the firm’s historical selections.

Key Points

• Architecture: three-step hybrid decision-support system
• Extended AHP (incorporating organizational learning patterns) to produce expert-validated criteria weights (consistency ratio CR = 0.04) and Linear Programming (LP) for portfolio optimization.
• Machine learning models (Transformers, TabNet, Neural Oblivious Decision Ensembles / NODE) augmented with expert knowledge through mechanisms such as attention modulation, split-criterion weighting, and differentiable tree regularization.
• AHP-Stacking meta-ensemble that adaptively balances expert-derived signals and data-driven patterns.
• Performance: 97.5% accuracy; 96.9% F1; AUC-ROC = 0.989. Reported as ~25% improvement versus baseline methods.
• Economic outcome: retrospective backtest suggests a 68.2% improvement in portfolio value — an estimated USD 83.5M incremental value based on the enterprise’s realized financial outcomes.
• Reliability: AHP criteria weights show high internal consistency (CR = 0.04), indicating dependable expert input.
• Limitations: evaluation is retrospective and restricted to a single enterprise; generalizability across firms, sectors, and geographies remains untested.

Data & Methods

• Data
  • Enterprise-level dataset of N = 10,000 completed software projects.
  • Dataset claimed to have “verified real-world statistical characteristics” (presumably verified distributions/quality checks), but from one firm only.
• Extended AHP
  • Integrates organizational learning patterns into the traditional AHP pairwise weighting process.
  • Resulting criteria weights validated by experts; consistency ratio = 0.04 (well below common 0.10 threshold).
• Portfolio optimization
  • Linear Programming used to translate project scores/weights into optimized portfolios under constraints.
• Machine learning architecture
  • Models: Transformers, TabNet, NODE (Neural Oblivious Decision Ensembles).
  • Expert knowledge incorporated by:
    • Attention modulation (biasing attention toward expert-prioritized features),
    • Split-criterion weighting (in tree-like models),
    • Differentiable tree regularization (to encourage structures consistent with expert rules).
• Hybrid ensemble
  • AHP-Stacking meta-classifier that blends expert-derived predictors and data-driven model outputs; adaptive weighting during training.
• Evaluation
  • Retrospective testing on same enterprise dataset; performance compared to baseline methods (details of baselines not listed in summary).
  • Economic impact estimated by mapping selected portfolios to realized financial performance (post factum).

Implications for AI Economics

• Firm-level productivity and value creation
  • The reported 68.2% portfolio-value uplift (USD 83.5M) suggests significant potential returns from hybrid human+AI decision systems in project selection and capital allocation.
  • If generalizable, such systems can materially improve resource allocation efficiency inside firms, increasing ROI on R&D and product portfolios.
• Complementarity of experts and ML
  • Demonstrates that embedding structured expert judgment into ML models can improve predictive accuracy and transparency — important for adoption in managerial settings where interpretability and trust matter.
  • Hybrid designs may reduce reliance on pure data-driven models when historical data are limited, noisy, or nonstationary.
• Adoption economics and barriers
  • High ex-post gains in one firm do not guarantee similar returns elsewhere: transaction costs, organizational fit, data availability, and implementation complexity may reduce realized benefits.
  • Upfront costs: expert elicitation, model integration, and optimization tooling; need cost–benefit and scaling analyses.
• Research and policy directions
  • Need for external validation: randomized controlled deployments or multi-firm retrospective studies to estimate average treatment effects and distribution of gains.
  • Evaluate robustness to domain shift: how performance decays across industries, firm size, and changing project technologies.
  • Distributional effects: potential shifts in managerial roles, changes in hiring/training needs, and concentration of decision-making capability.
  • Measurement caution: retrospective "post factum" value attribution risks overestimating causal gains (selection bias, confounding, or overfitting). Economists should prefer prospective experiments or careful counterfactual construction.
• Practical suggestions for economists and practitioners
  • Replicate study across multiple firms and industries; run randomized rollouts or phased A/B tests to estimate causal impact on portfolio returns.
  • Perform sensitivity analyses: alternate baselines, cost accounting for implementation, and stress tests for model misspecification or adversarial/structural change.
  • Quantify adoption thresholds: minimum data size, expert availability, and institutional conditions under which hybrid systems outperform existing processes.
  • Assess long-term effects on labor and governance: how decision autonomy, incentives, and accountability evolve when expert judgment is codified into ML pipelines.

Limitations to keep in mind: single-enterprise retrospective evaluation, potential overfitting to firm idiosyncrasies, unclear baseline/model-selection details, and possible optimism in backtested economic gains. Further multi-site and experimental work is required before generalizing the claimed economic impacts.