Explainable AI outperforms sales staff at predicting win/loss of complex configure‑to‑order offers — achieving 83.9% balanced accuracy overall versus 63.0% for human estimates — and provides SHAP explanations practitioners found understandable and actionable, though expert evaluation was limited to a small, selective sample.

Main Finding

An XGBoost model using CRM and CPQ data for configure-to-order industrial equipment predicted sales offer win/loss substantially better than experienced sales staff (balanced accuracy 83.9% on the full dataset, 74.1% on a regional subset vs. expert benchmark 63.0% and 54.4%, respectively). SHAP explanations identified key drivers (notably a PRICE FEATURE #1 and derived price-ratio features) and were judged by sales experts to be understandable and actionable, although some model-important features were unintuitive to experts.

Key Points

• Data: 439 finished sales offers after cleaning (1062 initial offers with CPQ+CRM; 615 had outcomes before filtering). Samples came from two countries: country A (n=146) and country B (n=293). Overall win rate ~33% (both countries) and ~56% in country A subset.
• Model & performance:
  • Algorithm: XGBoost (XGBClassifier, native/default parameters, random_state=66).
  • Preprocessing: categorical one-hot encoding; numerical standardization.
  • Train/test split: 90/10 random split (seed=210).
  • Metrics: balanced accuracy used to address class imbalance.
  • Results: balanced accuracy 83.9% (full dataset), 74.1% (country A-specific model).
• Human benchmark:
  • Salespeople record a win-probability in CRM; binarized (>50% = win) and used as expert baseline.
  • Expert baseline performance: balanced accuracy 63.0% (full), 54.4% (country A).
• Explainability:
  • SHAP (Shapley additive explanations) used to produce global (mean absolute SHAP) and local (beeswarm / per-case) explanations.
  • PRICE FEATURE #1 had the largest SHAP impact; a price-ratio (PRICE FEATURE #2 = PRICE1 / PRICE3) and other price-related features were also important. High values of PRICE FEATURE #1 increased predicted win probability.
• Expert evaluation:
  • Five business operations/sales experts reviewed 4 test cases (2 wins, 2 losses) with the model predictions and SHAP explanations, plus regional (country A) SHAP summaries.
  • Findings: partial alignment between model-important features and expert reasoning; some strong predictive features were considered unintuitive or unimportant by experts. Overall, experts found explanations understandable and useful for prioritization and offer modification.
• Open-data demonstration:
  • The authors also applied the pipeline to a public bank marketing dataset (after cleaning: n=30,488, 56 features) to demonstrate general model functionality.

Data & Methods

• Source: proprietary CRM + CPQ data from a configure-to-order industrial equipment vendor; sensitive fields anonymized and low-frequency categorical levels (≤5 occurrences) removed.
• Outcome: binary finished-offer result (won vs lost); unfinished offers were excluded.
• Feature set (post-encoding): product features (12 → 55 after encoding), organizational features (7 → 18), customer features (3 → 21), 3 extracted product features (→5), and 3 price features.
• Preprocessing:
  • Removed rows with very low-frequency categorical values.
  • One-hot encoding for categorical variables; standard scaling for numeric.
  • Excluded CRM entries where salespersons recorded exactly 50% predicted probability (overrepresented).
• Modeling:
  • XGBoost with native params, random_state=66.
  • Evaluation: single 90/10 train-test split (seed=210); balanced accuracy and confusion matrices reported.
• Explainability & evaluation:
  • SHAP for global and local feature importance and directional effects (mean absolute SHAP values, beeswarm plots).
  • Expert questionnaire: 5 respondents, presented selected top-N SHAP features (N chosen by SHAP threshold ≥0.29 mean) and asked to predict, rate feature importance, and rate agreement with explanations (Likert 1–5). Regional/global explanations presented in natural language plus arrow-quantified impacts.

Implications for AI Economics

• Productivity & allocation: High predictive accuracy implies AI can substantially improve lead prioritization and sales resource allocation in B2B CTO markets, raising expected returns per salesperson-hour by focusing effort on higher-probability offers.
• Pricing and offer design: Model-identified price-related features (PRICE FEATURE #1 and price ratio) being top predictors suggests firms can optimize configuring/pricing strategies to improve conversion; AI can surface counterintuitive pricing effects that human heuristics miss.
• Human-AI collaboration & adoption:
  • Explainable outputs (SHAP) increase interpretability and perceived actionability, facilitating adoption by sales teams and managers.
  • Partial misalignment between model drivers and expert intuition cautions that explainability does not guarantee acceptance—economic value depends on trust-building, user training, and integrating explanations into workflows.
• Labor and incentive effects:
  • Better predictions may shift commission structures and task allocation (e.g., fewer low-probability pursuits), with potential effects on salesforce workload, compensation fairness, and morale—requiring redesign of incentives.
• Value of explainability for decision support:
  • Explainable AI can make recommendations actionable (e.g., suggesting feature modifications), enabling firms to treat models not only as predictors but as prescriptive tools; this increases the marginal value of predictive models beyond forecast accuracy.
• Limits and risks that affect economic valuation:
  • Small proprietary sample (n=439) and geographic heterogeneity limit external validity; economic benefits should be validated with field experiments or A/B tests before large-scale deployment.
  • Excluding unfinished cases and overrepresented 50% expert entries may bias results; predictions may reflect product mix or historical pricing rather than causal levers.
  • Potential for models to reinforce historical biases (e.g., favoring certain customer types) that economically disadvantage segments—monitoring and constraints may be required.
• Research & practitioner priorities:
  • Economic assessment should move beyond predictive metrics to measure downstream outcomes: conversion lift, revenue impact, cost savings, and changes in salesperson behavior.
  • Randomized trials where teams use SHAP explanations to adjust offers (vs control) would quantify causal economic benefits and detect unintended effects on effort allocation and incentives.

Suggested next steps for researchers and practitioners: run randomized pilot deployments that measure revenue and effort reallocation, expand datasets across markets/products to test generalizability, and quantify cost–benefit including implementation, training, and governance costs.