A deterministic ML system triples a bank’s ability to flag risky IT changes: combining XGBoost with hybrid retrieval of historical change requests raises incident-catching from 20% to 63% while preserving per-decision explainability (ROC AUC 0.87).

Main Finding

SENTRY replaces subjective, questionnaire-based IT change risk scoring with a deterministic, explainable ML pipeline that integrates structured operational metadata and a hybrid retrieval-augmented-generation (RAG) layer distilled into a single scalar feature. On enterprise-scale change data, the system achieves ROC AUC = 0.87, ~85% overall accuracy, and raises detection of changes that later cause major incidents from ~20% (questionnaire baseline) to 63% — roughly a 3.25× improvement in recall for high-risk changes.

Key Points

• Architecture
  • Deterministic XGBoost classifier (28 features) + a hybrid RAG component that converts free-text change request content into a single deterministic scalar (hybrid_search_score).
  • System components: API backend, ITSM relational replica, application portfolio/dependency service, vector database with dense embeddings + lexical index.
• Hybrid RAG design
  • Five text fields (short description, description, implementation plan, test plan, backout plan) are concatenated and embedded.
  • Hybrid retrieval: semantic (dense cosine) + lexical (full-text) combined via Reciprocal Rank Fusion (RRF). α/β weighting favors lexical matches (paper uses α=0.4, β=0.6).
  • Top-k (default k=100) retrieved neighbors are aggregated into hybrid_search_score = sum(rrf_score × priority_points) over incident-bearing neighbors.
  • The scalar enters the XGBoost model as a single, explainable feature (contribution visible via SHAP-style pred_contribs).
• Modeling & explainability choices
  • XGBoost chosen because it is deterministic, performs well on tabular data, and natively exposes per-prediction feature contributions.
  • Avoids feeding raw embedding dimensions directly into the tree model to prevent opacity and domination of embedding features.
• Training & calibration
  • Labeled positives: ~183 change requests that caused P1/P2 incidents (Jan 2021–Jun 2026). Negative examples sampled to an ~80/20 class ratio.
  • Optuna (TPE) hyperparameter search (60 trials) with 5-fold stratified CV. Objective: maximize positive-class F1 with a small penalty on precision-recall imbalance.
  • Thresholds: High risk fixed at P ≥ 0.90; medium threshold tuned (searched 0.30–0.75) to balance precision and recall.
  • Class imbalance handled via scale_pos_weight computed from class ratio.
• Evaluation
  • Held-out test (n=203) confusion matrix: true positives 22, false negatives 13, false positives 18, true negatives 150.
  • Per-class metrics: Low risk F1 = 0.91; High/Medium F1 = 0.59 (precision 0.55, recall 0.63).
  • Hybrid_search_score is among the top features by gain (ranked 4th), confirming that free-text similarity adds predictive signal.
• Operational trade-offs
  • Higher recall for high-risk changes at the cost of lower precision — operationally acceptable in this domain where missing incidents is costly.
  • Determinism and per-prediction explainability prioritized to meet regulatory/audit requirements.

Data & Methods

• Data sources
  • ITSM database: change requests, configuration items, incident records.
  • Application portfolio/dependency service: business criticality, crown-jewel flags, RTO, user counts, dependency graph (used for blast-radius features).
  • Vector DB: dense embeddings of concatenated text fields + precomputed lexical index; denormalized to include incident metadata.
• Feature set
  • 28 normalized features (range [0,1]), grouped into application context, blast radius, incident history, similarity (hybrid_search_score), and change type.
• Hybrid retrieval specifics
  • Embeddings produced from five concatenated text fields.
  • Reciprocal Rank Fusion smoothing constant = 60; α=0.4, β=0.6 used to combine semantic and lexical ranks.
  • k = 100 neighbors by default; only neighbors with prior incidents contribute via priority-weighted sum.
  • Temporal leakage prevention: retrieval excludes the query change and filters to historical CRs opened before the target CR creation date.
• Model training & optimization
  • XGBoost hyperparameter search ranges provided (n_estimators, max_depth, learning_rate, min_child_weight, subsample, colsample_bytree, gamma, reg_alpha, reg_lambda).
  • Objective = F1_pos - 0.1 * |precision - recall|; 5-fold stratified CV during Optuna trials.
  • Threshold calibration selects medium threshold minimizing |precision - recall| with F1 tiebreaker.
• Evaluation metrics
  • ROC AUC = 0.87 on held-out test; reporting confusion matrix, per-class precision/recall/F1; comparison vs baseline questionnaire: 63% vs 20% detection of incident-causing changes.

Implications for AI Economics

• Value creation (reduced expected incident cost)
  • Higher recall on incident-causing changes (20% → 63%) implies a substantial reduction in undetected high-risk changes. If C is the average cost per major incident and expected incidents per period are I, a first-order expected-loss reduction ≈ I × C × (0.63 − 0.20). The organization should quantify I and C to estimate dollar benefits.
• Operational cost trade-offs (false positives)
  • Precision of 0.55 means additional reviewed changes that do not cause incidents; this increases reviewer time and potential process friction. Economic assessment must weigh reviewer-hour costs and opportunity costs against incident-cost savings.
  • The paper’s F1-with-balance objective explicitly encodes this trade-off; tuning the operating point (thresholds) allows an organization to move along the precision–recall frontier to meet local cost constraints.
• Adoption and regulatory compliance
  • Determinism and per-prediction explainability materially lower regulatory and audit barriers, reducing non-technical adoption friction and associated compliance costs.
  • Explainability also reduces organizational risk from opaque automation (easier to defend decisions to auditors and regulators).
• Implementation and maintenance costs
  • Engineering investment: embedding pipeline, vector DB maintenance (automating ingestion to avoid stale corpus), retraining pipeline, and integrating with enterprise ITSM and approval workflows.
  • Model degradation risk: technology landscape changes mean periodic retraining and corpus updates are required; costs of continuous monitoring and model governance should be budgeted.
• Incentive and behavioral effects
  • Moving from self-reported questionnaires to objective scoring reduces opportunities for requestor gaming and potential under-reporting; this can improve information quality and reduce downstream moral hazard.
  • Conversely, increased false positives could generate user pushback if not managed (trust erosion). Economic returns depend on maintaining reviewer trust and appropriate escalation policies.
• Generalizability and transfer costs
  • Methodology is portable but requires local retraining and re-calibration of features and normalization; deploying to another firm entails data mapping, retraining, and validation costs.
• Suggested economic analysis steps for deployment
  • Compute baseline incident rate and average incident cost; estimate expected incident-cost reduction under SENTRY’s recall improvement.
  • Estimate additional reviewer-hours triggered by SENTRY’s precision/false-positive rate and compute incremental reviewer cost.
  • Compare net benefit to implementation and ongoing operating costs (engineering + governance + data maintenance) to compute ROI and payback period.
  • Run sensitivity analysis over incident cost, incident frequency, and reviewer cost to understand robustness of investment decision.

If you want, I can: - produce a numeric back-of-envelope ROI template you can fill with your institution’s incident frequency/cost and reviewer costs, or - extract concise recommended next steps for operational rollout (monitoring, threshold tuning, retraining cadence, governance checklist).