A 'profit-per-decision' metric converts binary classifier performance into expected business value, and a Viability Assessment Framework helps firms evaluate ML projects before and after deployment; expert interviews and scenario tests with 11 companies suggest the approach improves alignment between technical teams and business stakeholders.

Main Finding

The authors introduce the Viability Assessment Framework centered on a new metric—profit-per-decision (ppd)—that links binary-classification model performance to economic outcomes and derives the estimator’s sampling distribution so uncertainty can be quantified via confidence intervals. Embedded into the ML life cycle, the framework produces formal, probabilistic “viability statements” that support go/no-go and deployment decisions across pre-development, post-development, and post-deployment stages. The framework was validated in the predictive/condition-based maintenance (PdM/CbM) domain via expert interviews, a scenario-based evaluation with 16 PdM experts from 11 companies, and an open-source web app.

Key Points

• Problem addressed

  • Organizations struggle to assess economic viability of ML projects because model performance, costs/benefits, and uncertainty are not linked systematically.
  • Two central questions: (1) How to combine economic perspective with ML performance? (2) How to support decisions across the ML lifecycle while accounting for uncertainty?

• Core contributions

  • Profit-per-decision (ppd): a cost–benefit metric for binary classification that maps confusion-matrix outcomes (TP, FP, TN, FN) into expected profit per model decision.
  • Statistical treatment: derivation of the sampling distribution of the ppd estimator (parametric) enabling construction of confidence intervals for ppd.
  • Viability Assessment Framework: uses ppd and its CI to
    • define minimal required model performance before development (pre-development),
    • evaluate sufficiency of developed models (post-development),
    • decide whether to continue use in production (post-deployment).
  • Produces interpretable viability statements of the form: “The project is viable with probability p given required performance q” (or vice versa).
  • Practical tooling and evaluation: demonstration in PdM/CbM use case, semi-structured interviews, scenario-based evaluation (including a real automotive manufacturer use case), and an open-source web app to facilitate expert interaction.

• Why PdM/CbM?

  • High and quantifiable economic impact of decisions (clear costs for FP/FN and benefits for TP/TN).
  • Typical small sample sizes increase uncertainty—highlighting the need for explicit uncertainty quantification.

• Relationship to prior work

  • Fills gaps in literature: most prior methods either focus on a single phase, ignore parts of the cost–benefit matrix, or do not jointly treat costs, performance, and uncertainty.
  • Complements existing performance-estimation workflows and MLOps/CRISP-ML(Q) practices rather than replacing them.

• Limitations noted by authors

  • Requires reliable cost/benefit elicitation from domain experts (subject to estimation error).
  • Parametric assumptions and stationarity assumptions can limit validity under severe distribution shift or model/data drift.
  • Applicability demonstrated for binary decisions; extension to multi-class or complex sequential decisions would require further work.
  • Small sample sizes remain a challenge—CIs may be wide, limiting decisive conclusions.

Data & Methods

• Systematic Literature Review (SLR)

  • Followed Webster & Watson (2002) and Wolfswinkel et al. (2013) frameworks to identify gaps at the intersection of costs/benefits, uncertainty, and minimal/sufficient model performance across ML life-cycle phases.

• Theoretical / statistical development

  • Defined ppd as the expected monetary payoff associated with a single model decision using a cost–benefit matrix tied to confusion-matrix rates.
  • Modeled confusion-matrix counts probabilistically (test-set outcomes) and derived the sampling distribution of the ppd estimator to obtain parametric confidence intervals (authors prefer parametric CIs for robustness and simplicity).
  • Integrated ROC/threshold considerations, showing how decision thresholds map into ppd and how ROC-based analyses may be insufficient when economic trade-offs are explicit.

• Empirical evaluation

  • Demonstration: applied framework to a PdM/CbM use case where costs/benefits of maintenance decisions are quantifiable.
  • Socio-technical evaluation:
    • Semi-structured expert interviews to assess relevance, usability, and gaps.
    • Scenario-based evaluation: 16 PdM experts across 11 companies applied the framework to a real automotive use case.
  • Tooling: developed an open-source web application to allow experts to interactively define cost–benefit inputs, view ppd estimates with CIs, and produce viability statements.

Implications for AI Economics

• Enables monetized, probabilistic decision-making

  • Directly connects model performance to financial impact (NPV/ROI) via ppd, allowing organizations to compare ML investments on consistent economic grounds.
  • Produces confidence intervals for expected profit per decision, converting statistical uncertainty into a financial risk statement—useful for risk-averse resource allocation.

• Better project selection and prioritization

  • Use pre-development capability to compute minimal required performance given budgeted costs, expected benefits, and acceptable uncertainty—helps screen and prioritize ML projects earlier and more objectively.
  • Supports cost-sensitive prioritization consistent with corporate ROI/NPV objectives rather than purely technical metrics (accuracy/AUC).

• Integrates with MLOps and governance

  • Provides a quantitative gate for deployment and model replacement decisions: deploy/keep only if ppd (with CI) meets the economic threshold at the required confidence level.
  • Can be embedded in CI/CD or MLOps pipelines to automate economically informed deployment policies.

• Improves communication between business and technical roles

  • Translating confusion-matrix outcomes into monetary terms and probability statements helps domain experts, managers, and engineers share a common decision vocabulary, reducing "fear of the unknown."

• Caveats for economic modeling and policy

  • Sensitivity to cost/benefit inputs: poor or biased estimates of FP/FN costs or TP/TN benefits will propagate into misleading viability statements—robust elicitation and sensitivity analysis are necessary.
  • Stationarity and drift: the ppd CIs reflect sampling uncertainty, not model misspecification or distributional shift; organizations must monitor deployed models and recalibrate the framework under drift.
  • Distributional assumptions: the parametric CI derivation simplifies computation and interpretability, but non-parametric checks (e.g., bootstrap) or Bayesian alternatives may be required in small-sample or heavy-tailed contexts.

Practical recommendations (for managers and AI economists) - Elicit and document cost–benefit values with domain experts; accompany these with sensitivity analyses. - Use the framework pre-development to screen projects and set realistic performance targets tied to economic objectives. - Integrate ppd + CI checks as deployment gates in MLOps pipelines to make probabilistic deployment/rollback decisions. - Combine the framework with continuous monitoring to capture drift and re-evaluate viability over time. - For research: extend ppd to multi-class settings, sequential decisions, and to incorporate epistemic/predictive uncertainty (e.g., Bayesian variants) and evaluate performance under distribution shift.