1. Development
Fit preprocessing and tune XGBoost only inside development folds. Class weighting and feature selection are development operations.
Leakage-resistant evaluation
Validation methodology
The research design separates development, threshold selection and final evaluation. A high AUROC alone is insufficient for an early-warning claim.
2. OOF prediction
Generate out-of-fold probabilities to compare configurations, assess calibration and select the high-sensitivity operating threshold.
3. Locked test
Freeze model identity and threshold before evaluation on held-out or external cohorts.
Primary operating-point rule
\[T_{watch}=\arg\max_T\mathrm{Specificity}(T)\quad\mathrm{s.t.}\quad\mathrm{Sensitivity}(T)\ge0.98\]
The final report must give specificity, PPV, NPV and alert burden at this same locked threshold—not at a retrospectively chosen test-set threshold.
Metrics
| Domain | Measures | Why it matters |
|---|---|---|
| Discrimination | AUROC, AUPRC | AUPRC is especially informative when SAP is the minority class. |
| Operating point | Sensitivity, specificity, PPV, NPV, Fβ | Shows the actual cost of targeting very high sensitivity. |
| Calibration | Brier score, slope, intercept, reliability curve | A ranked model can still give badly biased absolute probabilities. |
| Uncertainty | Bootstrap confidence intervals | Prevents over-reading point estimates from modest cohorts. |
| Utility | Decision-curve analysis, alert burden | Explores whether a threshold may add net benefit, without implying clinical approval. |
Multi-cohort validation
The preferred design is leave-one-cohort-out or explicitly external validation. Cohort identity is used for splitting and reporting, not as a shortcut predictor.
\[\text{Train on cohorts }\{C_1,\ldots,C_{k-1}\}\;\longrightarrow\;\text{Test on untouched }C_k\]
Performance should be shown for each hospital/cohort before any pooled summary. The two Guilin sources require particular caution because they may contain overlapping patients.
Early deterioration extension
\[Y_{t,h}=\mathbf{1}\{\text{new deterioration occurs in }(t,t+h]\}\]
For time-stamped EHR cohorts, predictions can be evaluated at admission, 6 h, 12 h and 24 h. Events already present at time \(t\) must not receive “early warning” credit.
Research boundary. Retrospective validation does not establish a bedside escalation rule. Prospective clinical validation, governance and regulatory assessment would still be required.