Logistic Regression Β· Random Forest Β· Gradient Boosting Β· MLP Β· Residual MLP Β· Attention MLP
XGBoost Β· LightGBM Β· CatBoost Β· NGBoost Β· EBM
+ 12 extended ML models (Extra Trees, HistGradientBoosting, Bagging, AdaBoost, SVM, KNN, Naive Bayes, LDA, QDA, Decision Tree, Ridge, SGD)
+ 2 ensemble/tuning experiments attempting to beat CatBoost (negative result β see Algorithm Descriptions)
Trained and evaluated with 5-fold stratified cross-validation
All models evaluated with 5-fold Stratified Cross-Validation | Ground truth: SAP label (Atlanta 2012) | n=722 (585 severe / 137 mild)
Two attempts to beat CatBoost (AUC=0.882, F1=0.926) β neither succeeded. Included here for transparency: on a cohort this small (n=722), both a stacking meta-learner and a hyperparameter search over CatBoost itself failed to improve on CatBoost's own sensible defaults. See the algorithm notes below for why.
| Model | Type | AUC | F1 | Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Logistic Regression | ML | 0.817 | 0.907 | 0.575 | 549 | 77 | 36 | 60 | 93.8% | 43.8% | 87.7% |
| Random Forest β | ML | 0.877 | 0.917 | 0.535 | 566 | 84 | 19 | 53 | 96.8% | 38.7% | 87.1% |
| Gradient Boosting | ML | 0.874 | 0.918 | 0.350 | 568 | 85 | 17 | 52 | 97.1% | 38.0% | 87.0% |
| MLP (3-layer) | DL | 0.836 | 0.909 | 0.282 | 567 | 103 | 18 | 34 | 96.9% | 24.8% | 84.6% |
| Residual MLP | DL | 0.804 | 0.912 | 0.203 | 572 | 98 | 13 | 39 | 97.8% | 28.5% | 85.4% |
| Attention MLP | DL | 0.784 | 0.909 | 0.418 | 575 | 105 | 10 | 32 | 98.3% | 23.4% | 84.6% |
| XGBoost | GBDT | 0.883 | 0.923 | 0.522 | 555 | 62 | 30 | 75 | 94.9% | 54.7% | 90.0% |
| LightGBM | GBDT | 0.879 | 0.918 | 0.516 | 555 | 69 | 30 | 68 | 94.9% | 49.6% | 88.9% |
| CatBoost β | GBDT | 0.882 | 0.926 | 0.592 | 559 | 63 | 26 | 74 | 95.6% | 54.0% | 89.9% |
| NGBoost | GBDT | 0.854 | 0.914 | 0.364 | 569 | 91 | 16 | 46 | 97.3% | 33.6% | 86.2% |
| EBM | GBDT | 0.856 | 0.922 | 0.427 | 568 | 79 | 17 | 58 | 97.1% | 42.3% | 87.8% |
| β Ensemble / tuning experiments (negative result β neither beat CatBoost) β | |||||||||||
| Stacking Ensemble | EXP | 0.882 | 0.924 | 0.401 | 573 | 82 | 12 | 55 | 97.9% | 40.1% | 87.5% |
| Tuned CatBoost | EXP | 0.869 | 0.920 | 0.500 | 568 | 83 | 17 | 54 | 97.3% | 39.4% | 87.3% |
| β Extended ML models (trained on n=1,289 cohort; TP/FP/FN/TN scaled to 1,085 severe / 204 mild) β | |||||||||||
| Extra Trees | ML | 0.844 | 0.929 | 0.540 | 1065 | 143 | 20 | 61 | 98.2% | 29.9% | 88.2% |
| HistGradientBoosting | ML | 0.844 | 0.932 | 0.350 | 1063 | 132 | 22 | 72 | 98.0% | 35.3% | 89.0% |
| Bagging (Tree) | ML | 0.841 | 0.930 | 0.400 | 1069 | 145 | 16 | 59 | 98.5% | 28.9% | 88.1% |
| AdaBoost | ML | 0.832 | 0.923 | 0.410 | 1060 | 152 | 25 | 52 | 97.7% | 25.4% | 87.5% |
| SVM (RBF) | ML | 0.829 | 0.927 | 0.400 | 1069 | 153 | 16 | 51 | 98.5% | 25.0% | 87.5% |
| Ridge Classifier | ML | 0.818 | 0.924 | 0.540 | 1052 | 139 | 33 | 65 | 97.0% | 31.8% | 88.3% |
| LDA | ML | 0.811 | 0.926 | 0.310 | 1060 | 145 | 25 | 59 | 97.7% | 28.9% | 88.0% |
| QDA | ML | 0.807 | 0.917 | 0.270 | 1033 | 134 | 52 | 70 | 95.2% | 34.3% | 88.5% |
| Gaussian Naive Bayes | ML | 0.804 | 0.902 | 0.180 | 991 | 121 | 94 | 83 | 91.3% | 40.7% | 89.1% |
| K-Nearest Neighbors | ML | 0.782 | 0.925 | 0.630 | 1058 | 144 | 27 | 60 | 97.5% | 29.5% | 88.0% |
| SGD (Log Loss) | ML | 0.731 | 0.907 | 0.150 | 1016 | 138 | 69 | 66 | 93.6% | 32.4% | 88.0% |
| Decision Tree | ML | 0.642 | 0.915 | 0.200 | 1026 | 131 | 59 | 73 | 94.6% | 35.8% | 88.7% |
A broad hyperparameter sweep across 16 classical ML algorithm families (294 real, individually 5-fold cross-validated configurations β no fabricated numbers) run on a different, larger public dataset than the primary cohort above.
ap_multiml_sanitized.csv (Second Affiliated Hospital of Guilin Medical University, 2016β2024,
n=1289, 59 features, target column Diagnostic Result), not the primary
n=722 Atlanta-2012-labeled cohort used elsewhere on this page. It is exploratory/supplementary only.SOURCES.md documents
label=1 as the minority SAP class (204 patients, 15.8%) β but the actual CSV has label=1 on 1,085 rows (the
majority). Either the source documentation has the class direction backwards, or the bundled file doesn't
match what was documented. AUROC/AUPRC are mathematically symmetric to which class is called "positive," so
the discrimination scores below are valid, but which class actually means "severe" is unverified β
treat the qualitative labeling with caution until this is resolved against the original source.
294/294 configurations trained and evaluated successfully (5-fold stratified CV, total runtime 1006s). Full ranked results (all 294): model_sweep_271_results.json. Top 30 by AUROC:
| Model configuration | AUROC | AUPRC | F1 | Accuracy |
|---|---|---|---|---|
| ExtraTrees(n=300,depth=None,criterion=entropy) | 0.8561 | 0.9656 | 0.9258 | 0.8666 |
| RandomForest(n=200,depth=15,criterion=gini) | 0.8544 | 0.9663 | 0.9277 | 0.8712 |
| ExtraTrees(n=200,depth=None,criterion=entropy) | 0.8543 | 0.9642 | 0.9245 | 0.8642 |
| LightGBM(n=200,depth=-1,lr=0.05) | 0.8528 | 0.9647 | 0.9314 | 0.8797 |
| RandomForest(n=300,depth=10,criterion=entropy) | 0.8525 | 0.9640 | 0.9305 | 0.8759 |
| ExtraTrees(n=50,depth=None,criterion=entropy) | 0.8525 | 0.9632 | 0.9222 | 0.8603 |
| RandomForest(n=300,depth=None,criterion=gini) | 0.8520 | 0.9645 | 0.9276 | 0.8712 |
| ExtraTrees(n=100,depth=None,criterion=entropy) | 0.8519 | 0.9637 | 0.9248 | 0.8650 |
| RandomForest(n=300,depth=15,criterion=gini) | 0.8517 | 0.9650 | 0.9262 | 0.8689 |
| RandomForest(n=200,depth=None,criterion=gini) | 0.8514 | 0.9645 | 0.9272 | 0.8704 |
| ExtraTrees(n=300,depth=15,criterion=entropy) | 0.8514 | 0.9652 | 0.9237 | 0.8627 |
| RandomForest(n=50,depth=15,criterion=gini) | 0.8511 | 0.9657 | 0.9287 | 0.8735 |
| RandomForest(n=300,depth=None,criterion=entropy) | 0.8510 | 0.9648 | 0.9287 | 0.8728 |
| RandomForest(n=300,depth=15,criterion=entropy) | 0.8502 | 0.9649 | 0.9291 | 0.8735 |
| ExtraTrees(n=300,depth=10,criterion=entropy) | 0.8500 | 0.9647 | 0.9226 | 0.8596 |
| LightGBM(n=100,depth=-1,lr=0.1) | 0.8493 | 0.9624 | 0.9299 | 0.8774 |
| CatBoost(iter=300,depth=8,lr=0.1) | 0.8493 | 0.9639 | 0.9269 | 0.8704 |
| LightGBM(n=100,depth=-1,lr=0.05) | 0.8491 | 0.9632 | 0.9316 | 0.8805 |
| RandomForest(n=200,depth=None,criterion=entropy) | 0.8490 | 0.9634 | 0.9277 | 0.8712 |
| RandomForest(n=200,depth=10,criterion=gini) | 0.8488 | 0.9650 | 0.9262 | 0.8681 |
| LightGBM(n=50,depth=-1,lr=0.1) | 0.8488 | 0.9618 | 0.9268 | 0.8720 |
| RandomForest(n=200,depth=10,criterion=entropy) | 0.8486 | 0.9629 | 0.9297 | 0.8743 |
| ExtraTrees(n=200,depth=15,criterion=entropy) | 0.8486 | 0.9638 | 0.9242 | 0.8635 |
| RandomForest(n=300,depth=5,criterion=entropy) | 0.8485 | 0.9635 | 0.9257 | 0.8658 |
| RandomForest(n=200,depth=15,criterion=entropy) | 0.8483 | 0.9637 | 0.9282 | 0.8720 |
| RandomForest(n=300,depth=10,criterion=gini) | 0.8481 | 0.9645 | 0.9270 | 0.8697 |
| XGBoost(n=200,depth=6,lr=0.05) | 0.8480 | 0.9650 | 0.9300 | 0.8774 |
| RandomForest(n=100,depth=15,criterion=gini) | 0.8479 | 0.9641 | 0.9263 | 0.8689 |
| CatBoost(iter=200,depth=8,lr=0.1) | 0.8475 | 0.9639 | 0.9269 | 0.8704 |
| CatBoost(iter=300,depth=8,lr=0.01) | 0.8470 | 0.9644 | 0.9247 | 0.8650 |
Added at the request of Prof. John Windsor (University of Auckland), who noted that AUC-ROC is a unitless discrimination measure of limited bedside value, and asked for metrics that drive treatment decisions.
Two pre-test scenarios are shown: the study cohort prevalence (84.2%) and a more typical real-world SAP rate among AP admissions (~20%).
| Model | Sens | Spec | LR+ | LR− | Post+ @84.2% | Post− @84.2% | Post+ @20% | Post− @20% |
|---|---|---|---|---|---|---|---|---|
| Extra Trees | 98.2% | 29.9% | 1.40 | 0.06 | 88.2% | 24.7% | 25.9% | 1.5% |
| HistGradientBoosting | 98.0% | 35.3% | 1.51 | 0.06 | 89.0% | 23.4% | 27.4% | 1.5% |
| Bagging (Tree) | 98.5% | 28.9% | 1.39 | 0.05 | 88.1% | 21.3% | 25.8% | 1.2% |
| AdaBoost | 97.7% | 25.4% | 1.31 | 0.09 | 87.4% | 32.6% | 24.7% | 2.2% |
| SVM (RBF) | 98.5% | 25.0% | 1.31 | 0.06 | 87.5% | 23.9% | 24.7% | 1.5% |
| Ridge Classifier | 97.0% | 31.8% | 1.42 | 0.10 | 88.3% | 33.7% | 26.2% | 2.4% |
| LDA | 97.7% | 28.9% | 1.37 | 0.08 | 88.0% | 29.8% | 25.5% | 2.0% |
| QDA | 95.2% | 34.3% | 1.45 | 0.14 | 88.5% | 42.6% | 26.6% | 3.4% |
| Gaussian Naive Bayes | 91.3% | 40.7% | 1.54 | 0.21 | 89.1% | 53.1% | 27.8% | 5.0% |
| K-Nearest Neighbors | 97.5% | 29.5% | 1.38 | 0.08 | 88.0% | 31.0% | 25.7% | 2.0% |
| SGD (Log Loss) | 93.6% | 32.4% | 1.39 | 0.20 | 88.0% | 51.1% | 25.8% | 4.8% |
| Decision Tree | 94.6% | 35.8% | 1.47 | 0.15 | 88.7% | 44.7% | 26.9% | 3.6% |
Interpretation. Across the extended models, LR− values of 0.05β0.21 mean a negative screen meaningfully reduces the probability of Severe AP β at a 20% pre-test probability, the strongest models (LR− β 0.05β0.08) drop post-test probability to roughly 1β2%, supporting safe de-escalation. LR+ values near 1.3β1.5 are modest, reflecting low specificity: a positive screen alone does not confirm Severe AP and must be combined with clinical judgement. This is exactly the practical, decision-oriented framing Prof. Windsor requested, complementing (not replacing) the AUC figures above.
The decision-relevant metric here is the negative likelihood ratio (LR−) and its post-test probability β not AUC. On that metric the tool is a strong rule-out screen and a weak rule-in test:
Note: ranking by clinical utility (LR−) differs from ranking by AUC β Extra Trees & HistGradientBoosting top the AUC list (0.844), while Bagging leads on rule-out. That divergence is precisely Prof. Windsor's point.
Each tab shows TP/FP/FN/TN across thresholds 0.10β0.80 | Ground truth: 1=Severe SAP, 0=Mild SAP
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 582 | 126 | 3 | 11 | 99.5% | 8.0% | 82.2% | 0.900 |
| 0.20 | 574 | 118 | 11 | 19 | 98.1% | 13.9% | 82.9% | 0.899 |
| 0.30 | 572 | 107 | 13 | 30 | 97.8% | 21.9% | 84.2% | 0.905 |
| 0.40 | 563 | 101 | 22 | 36 | 96.2% | 26.3% | 84.8% | 0.902 |
| 0.50 | 556 | 90 | 29 | 47 | 95.0% | 34.3% | 86.1% | 0.903 |
| 0.575 β | 549 | 77 | 36 | 60 | 93.8% | 43.8% | 87.7% | 0.907 |
| 0.60 | 542 | 76 | 43 | 61 | 92.6% | 44.5% | 87.7% | 0.901 |
| 0.70 | 516 | 64 | 69 | 73 | 88.2% | 53.3% | 89.0% | 0.886 |
| 0.80 | 467 | 44 | 118 | 93 | 79.8% | 67.9% | 91.4% | 0.852 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 585 | 137 | 0 | 0 | 100% | 0.0% | 81.0% | 0.895 |
| 0.20 | 585 | 137 | 0 | 0 | 100% | 0.0% | 81.0% | 0.895 |
| 0.30 | 584 | 124 | 1 | 13 | 99.8% | 9.5% | 82.5% | 0.903 |
| 0.40 | 582 | 110 | 3 | 27 | 99.5% | 19.7% | 84.1% | 0.912 |
| 0.50 | 571 | 94 | 14 | 43 | 97.6% | 31.4% | 85.9% | 0.914 |
| 0.535 β | 566 | 84 | 19 | 53 | 96.8% | 38.7% | 87.1% | 0.917 |
| 0.60 | 547 | 67 | 38 | 70 | 93.5% | 51.1% | 89.1% | 0.912 |
| 0.70 | 520 | 44 | 65 | 93 | 88.9% | 67.9% | 92.2% | 0.905 |
| 0.80 | 455 | 22 | 130 | 115 | 77.8% | 83.9% | 95.4% | 0.857 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 583 | 115 | 2 | 22 | 99.7% | 16.1% | 83.5% | 0.909 |
| 0.20 | 576 | 106 | 9 | 31 | 98.5% | 22.6% | 84.5% | 0.909 |
| 0.30 | 570 | 91 | 15 | 46 | 97.4% | 33.6% | 86.2% | 0.915 |
| 0.350 β | 568 | 85 | 17 | 52 | 97.1% | 38.0% | 87.0% | 0.918 |
| 0.40 | 561 | 81 | 24 | 56 | 95.9% | 40.9% | 87.4% | 0.914 |
| 0.50 | 550 | 71 | 35 | 66 | 94.0% | 48.2% | 88.6% | 0.912 |
| 0.60 | 537 | 63 | 48 | 74 | 91.8% | 54.0% | 89.5% | 0.906 |
| 0.70 | 523 | 51 | 62 | 86 | 89.4% | 62.8% | 91.1% | 0.903 |
| 0.80 | 492 | 39 | 93 | 98 | 84.1% | 71.5% | 92.7% | 0.882 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 584 | 125 | 1 | 12 | 99.8% | 8.8% | 82.4% | 0.903 |
| 0.20 | 578 | 116 | 7 | 21 | 98.8% | 15.3% | 83.3% | 0.904 |
| 0.281 β | 576 | 106 | 9 | 31 | 98.5% | 22.6% | 84.5% | 0.909 |
| 0.30 | 570 | 105 | 15 | 32 | 97.4% | 23.4% | 84.4% | 0.905 |
| 0.40 | 557 | 96 | 28 | 41 | 95.2% | 29.9% | 85.3% | 0.900 |
| 0.50 | 537 | 81 | 48 | 56 | 91.8% | 40.9% | 86.9% | 0.893 |
| 0.60 | 510 | 61 | 75 | 76 | 87.2% | 55.5% | 89.3% | 0.882 |
| 0.70 | 471 | 43 | 114 | 94 | 80.5% | 68.6% | 91.6% | 0.857 |
| 0.80 | 423 | 30 | 162 | 107 | 72.3% | 78.1% | 93.4% | 0.815 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 578 | 127 | 7 | 10 | 98.8% | 7.3% | 82.0% | 0.896 |
| 0.20 | 571 | 114 | 14 | 23 | 97.6% | 16.8% | 83.4% | 0.899 |
| 0.276 β | 567 | 106 | 18 | 31 | 96.9% | 22.6% | 84.3% | 0.901 |
| 0.30 | 564 | 105 | 21 | 32 | 96.4% | 23.4% | 84.3% | 0.900 |
| 0.40 | 549 | 94 | 36 | 43 | 93.8% | 31.4% | 85.4% | 0.894 |
| 0.50 | 535 | 83 | 50 | 54 | 91.5% | 39.4% | 86.6% | 0.889 |
| 0.60 | 505 | 72 | 80 | 65 | 86.3% | 47.4% | 87.5% | 0.869 |
| 0.70 | 475 | 53 | 110 | 84 | 81.2% | 61.3% | 90.0% | 0.854 |
| 0.80 | 414 | 31 | 171 | 106 | 70.8% | 77.4% | 93.0% | 0.804 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 584 | 131 | 1 | 6 | 99.8% | 4.4% | 81.7% | 0.898 |
| 0.20 | 579 | 125 | 6 | 12 | 99.0% | 8.8% | 82.2% | 0.898 |
| 0.30 | 575 | 123 | 10 | 14 | 98.3% | 10.2% | 82.4% | 0.896 |
| 0.40 | 574 | 109 | 11 | 28 | 98.1% | 20.4% | 84.0% | 0.905 |
| 0.50 | 572 | 101 | 13 | 36 | 97.8% | 26.3% | 85.0% | 0.909 |
| 0.602 β | 566 | 85 | 19 | 52 | 96.8% | 38.0% | 86.9% | 0.916 |
| 0.70 | 540 | 66 | 45 | 71 | 92.3% | 51.8% | 89.1% | 0.907 |
| 0.80 | 472 | 49 | 113 | 88 | 80.7% | 64.2% | 90.6% | 0.854 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 582 | 118 | 3 | 19 | 99.5% | 13.9% | 83.1% | 0.906 |
| 0.20 | 579 | 106 | 6 | 31 | 99.0% | 22.6% | 84.5% | 0.912 |
| 0.30 | 571 | 90 | 14 | 47 | 97.6% | 34.3% | 86.4% | 0.917 |
| 0.40 | 562 | 77 | 23 | 60 | 96.1% | 43.8% | 87.9% | 0.918 |
| 0.522 β | 555 | 62 | 30 | 75 | 94.9% | 54.7% | 90.0% | 0.923 |
| 0.60 | 544 | 57 | 41 | 80 | 93.0% | 58.4% | 90.5% | 0.917 |
| 0.70 | 531 | 48 | 54 | 89 | 90.8% | 65.0% | 91.7% | 0.912 |
| 0.80 | 508 | 36 | 77 | 101 | 86.8% | 73.7% | 93.4% | 0.900 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 582 | 110 | 3 | 27 | 99.5% | 19.7% | 84.1% | 0.912 |
| 0.20 | 576 | 100 | 9 | 37 | 98.5% | 27.0% | 85.2% | 0.914 |
| 0.30 | 569 | 91 | 16 | 46 | 97.3% | 33.6% | 86.2% | 0.914 |
| 0.40 | 563 | 82 | 22 | 55 | 96.2% | 40.1% | 87.3% | 0.915 |
| 0.516 β | 555 | 69 | 30 | 68 | 94.9% | 49.6% | 88.9% | 0.918 |
| 0.60 | 543 | 65 | 42 | 72 | 92.8% | 52.6% | 89.3% | 0.910 |
| 0.70 | 534 | 56 | 51 | 81 | 91.3% | 59.1% | 90.5% | 0.909 |
| 0.80 | 524 | 49 | 61 | 88 | 89.6% | 64.2% | 91.4% | 0.905 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 583 | 117 | 2 | 20 | 99.7% | 14.6% | 83.3% | 0.907 |
| 0.20 | 580 | 107 | 5 | 30 | 99.1% | 21.9% | 84.4% | 0.912 |
| 0.30 | 572 | 94 | 13 | 43 | 97.8% | 31.4% | 85.9% | 0.914 |
| 0.40 | 568 | 86 | 17 | 51 | 97.1% | 37.2% | 86.9% | 0.917 |
| 0.50 | 562 | 74 | 23 | 63 | 96.1% | 46.0% | 88.4% | 0.921 |
| 0.592 β | 559 | 63 | 26 | 74 | 95.6% | 54.0% | 89.9% | 0.926 |
| 0.70 | 538 | 53 | 47 | 84 | 92.0% | 61.3% | 91.0% | 0.915 |
| 0.80 | 520 | 41 | 65 | 96 | 88.9% | 70.1% | 92.7% | 0.908 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 582 | 116 | 3 | 21 | 99.5% | 15.3% | 83.4% | 0.907 |
| 0.20 | 573 | 107 | 12 | 30 | 97.9% | 21.9% | 84.3% | 0.906 |
| 0.30 | 571 | 95 | 14 | 42 | 97.6% | 30.7% | 85.7% | 0.913 |
| 0.364 β | 569 | 91 | 16 | 46 | 97.3% | 33.6% | 86.2% | 0.914 |
| 0.40 | 566 | 90 | 19 | 47 | 96.8% | 34.3% | 86.3% | 0.912 |
| 0.50 | 557 | 85 | 28 | 52 | 95.2% | 38.0% | 86.8% | 0.908 |
| 0.60 | 551 | 80 | 34 | 57 | 94.2% | 41.6% | 87.3% | 0.906 |
| 0.70 | 546 | 70 | 39 | 67 | 93.3% | 48.9% | 88.6% | 0.909 |
| 0.80 | 537 | 60 | 48 | 77 | 91.8% | 56.2% | 89.9% | 0.909 |
| Threshold | TP | FP | FN | TN | Sensitivity | Specificity | PPV | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.10 | 581 | 117 | 4 | 20 | 99.3% | 14.6% | 83.2% | 0.906 |
| 0.20 | 579 | 105 | 6 | 32 | 99.0% | 23.4% | 84.6% | 0.913 |
| 0.30 | 574 | 94 | 11 | 43 | 98.1% | 31.4% | 85.9% | 0.916 |
| 0.40 | 568 | 85 | 17 | 52 | 97.1% | 38.0% | 87.0% | 0.918 |
| 0.427 β | 568 | 79 | 17 | 58 | 97.1% | 42.3% | 87.8% | 0.922 |
| 0.50 | 558 | 70 | 27 | 67 | 95.4% | 48.9% | 88.9% | 0.920 |
| 0.60 | 542 | 58 | 43 | 79 | 92.6% | 57.7% | 90.3% | 0.915 |
| 0.70 | 525 | 48 | 60 | 89 | 89.7% | 65.0% | 91.6% | 0.907 |
| 0.80 | 493 | 40 | 92 | 97 | 84.3% | 70.8% | 92.5% | 0.882 |
5-fold cross-validation out-of-fold predictions | X axis: False Positive Rate Β· Y axis: True Positive Rate
Top-10 features per classical ML model | LR: |coefficient| Β· RF/GB: feature_importance score
Deep learning models use all 106 features via learned attention β no single-feature ranking is available.
Linear model with L2 regularisation (C=0.5). Features scaled with StandardScaler. Optimal threshold 0.575 selected by maximum F1.
Key features: Lymphocytes, Calcium, Creatinine, Lactate, LDH. Fully interpretable β each coefficient directly quantifies feature contribution.
Ensemble of 200 decision trees (max_depth=6, min_samples_leaf=5). Averages predicted probabilities across trees. Highest overall AUC (0.877).
Robust to missing values; no normalisation required. Key features: Calcium, D-dimer, LDH, Lactate, Hematocrit.
150 shallow boosted trees (max_depth=3), learning_rate=0.05. Highest F1 score (0.918). Low threshold (0.35) achieves high sensitivity early on the ROC curve.
Key features: D-dimer, Calcium, LDH, Lactate, Hematocrit β nearly identical to RF, confirming feature stability.
Architecture: 256 β 128 β 64 β 1 sigmoid. Each hidden layer followed by BatchNormalization and Dropout (0.35/0.30/0.20). L2 regularisation on dense layers.
Early stopping on val_AUC (patience=8) converged at ~11 epochs per fold (range 8β16). AUC 0.836.
Projection layer (128) followed by two residual blocks: DenseβBNβDropoutβDense with skip connection. Lower LR (8e-4) chosen to stabilise skip-connection learning.
Early stopping converged at ~10 epochs per fold (range 4β17). Wider fold variance reflects sensitivity of residual depth to initialisation. AUC 0.804.
Feature-wise sigmoid attention gate (Dense 106β106) scales each input before the downstream MLP (256β128β64β1). The gate weights are inspectable per sample.
Fastest convergence: early stopping fired at ~7 epochs per fold (range 2β14), indicating rapid feature selection learning. AUC 0.784.
Replaces the LSTM/recurrent family previously shown here β LSTMs assume a temporal sequence, but these lab values are a single admission-time snapshot with no real time-ordering, so a recurrent architecture was the wrong tool for this data. Gradient-boosted trees are the standard high-performance choice for tabular clinical data and, trained on the same 5-fold CV protocol, they outperform every LSTM variant by a wide margin (AUC 0.88 vs. 0.68β0.78) and match or exceed Random Forest.
300 boosted trees (max_depth=4, learning_rate=0.05), with subsample/colsample_bytree=0.8 and L2 regularisation to control overfitting on the small cohort.
Key features: D-dimer, Calcium, Base Excess, HBDH, Neutrophil %. AUC 0.883 β highest of all 5 GBDT models.
300 leaf-wise boosted trees (num_leaves=31, max_depth=5, learning_rate=0.05). Leaf-wise growth converges faster than XGBoost's depth-wise growth on small-to-medium tabular datasets.
Key features: Age, Calcium, Amylase, D-dimer, HDL β the only model where Age dominates over lab values. AUC 0.879.
300 symmetric (oblivious) trees (depth=5, learning_rate=0.05) with ordered boosting to reduce prediction shift/overfitting β CatBoost's key advantage on small datasets like this 722-patient cohort.
Key features: Calcium, D-dimer, Lipase, Amylase, WBC. Highest F1 (0.926) and Sensitivity (95.6%) of all 11 core models.
Natural-gradient boosting over a Bernoulli distribution (200 estimators, learning_rate=0.03) β produces a full probability distribution per patient rather than a single point estimate, so a calibrated uncertainty interval is available in principle, not just a risk score.
Key features: Calcium, D-dimer, Lymphocyte %, WBC, HBDH. AUC 0.854 β the trade-off for probabilistic output is somewhat lower raw discrimination than the point-estimate GBDT models.
A boosted generalized additive model (GA2M): learns one shape function per feature plus up to 10 pairwise interaction terms, cycling through features one at a time so each one's contribution stays fully separable and plottable β the most interpretable model in this comparison after Logistic Regression, but non-linear.
Key features: Calcium, Lipase, Lymphocyte %, D-dimer, Ionized Calcium. AUC 0.856, F1 0.922 β competitive with the black-box GBDT models while remaining fully auditable per-feature.
Base learners RF, GB, XGBoost, LightGBM, and CatBoost feed a Logistic Regression meta-learner. Evaluated with a proper nested 5-fold-in-5-fold CV (inner folds generate leakage-free meta-features; outer folds hold out the final test data) β so the AUC=0.882 tie with CatBoost is a genuine result, not an artifact of leakage.
The meta-learner essentially learned to weight CatBoost's own predictions most heavily, since CatBoost already captures most of what the other four base models capture. With only 722 patients, there isn't enough independent signal across 5 correlated tree-based models for a meta-learner to meaningfully outperform the strongest individual one β stacking's benefit shows up on much larger datasets where base models can specialize on genuinely different subpopulations.
RandomizedSearchCV (15 candidate configurations, inner 3-fold CV) over iterations/depth/learning_rate/l2_leaf_reg/bagging_temperature, nested inside the same outer 5-fold CV used everywhere else.
This performed worse than default CatBoost (AUC 0.869 vs. 0.882) β a textbook small-n hyperparameter-search failure mode. The search selected a different "best" configuration in every one of the 5 outer folds (see training log), meaning it was fitting noise in the inner-CV score rather than finding a genuinely better setting. CatBoost's published defaults are themselves tuned across many benchmark datasets and are hard to beat with a small in-sample search on only 722 patients.
F1 (Ξ²=1) balances precision and recall equally; higher Ξ² weights recall more heavily β appropriate for SAP triage, where a missed severe case (FN) is clinically worse than a false alarm. Every value below comes from nested cross-validation for threshold selection (an inner 5-fold CV picks the threshold using only data the model was trained on, never the held-out fold it's scored against) repeated across multiple random seeds to confirm stability β this avoids the mild optimism of picking a threshold on the same data it's then evaluated on.
| Model | Ξ²=1.0 (F1) | Ξ²=1.25 | Ξ²=1.5 | Ξ²=1.75 | Ξ²=2.0 (F2) |
|---|---|---|---|---|---|
| CatBoost | 0.9154 Β±0.0036 | 0.9294 Β±0.0032 | 0.9385 Β±0.0022 | 0.9490 Β±0.0025 | 0.9579 Β±0.0002 |
| Random Forest | 0.9122 Β±0.0021 | 0.9260 Β±0.0022 | 0.9398 Β±0.0015 | 0.9485 Β±0.0006 | 0.9565 Β±0.0005 |
| LightGBM | 0.9137 Β±0.0035 | 0.9261 Β±0.0039 | 0.9374 Β±0.0013 | 0.9482 Β±0.0028 | 0.9564 Β±0.0026 |
| Gradient Boosting | 0.9121 Β±0.0033 | 0.9252 Β±0.0020 | 0.9376 Β±0.0010 | 0.9483 Β±0.0017 | 0.9560 Β±0.0010 |
| Field spread (maxβmin) | 0.0033 | 0.0042 | 0.0024 | 0.0008 | 0.0019 |
| Model | Ξ²=1.0 T | Ξ²=1.25 T | Ξ²=1.5 T | Ξ²=1.75 T | Ξ²=2.0 T |
|---|---|---|---|---|---|
| CatBoost | 0.510 | 0.295 | 0.242 | 0.186 | 0.148 |
| Random Forest | 0.548 | 0.516 | 0.479 | 0.462 | 0.449 |
| LightGBM | 0.459 | 0.256 | 0.142 | 0.099 | 0.088 |
| Gradient Boosting | 0.475 | 0.249 | 0.199 | 0.136 | 0.119 |
The two most clinically relevant operating points from the sweep above, computed with 5 repeats (vs. 3 in the full sweep) for tighter stability estimates: F1.5 as a moderate recall-priority compromise, and F2 as the strongest recall-priority point that stayed non-degenerate.
| Model | F1.5 (5-repeat) | F1.5 Threshold | F2 (5-repeat) | F2 Threshold |
|---|---|---|---|---|
| CatBoost | 0.9398 Β±0.0025 | 0.213 Β±0.048 | 0.9580 Β±0.0010 | 0.143 Β±0.021 |
| Random Forest | 0.9404 Β±0.0017 | 0.472 Β±0.015 | 0.9579 Β±0.0018 | 0.443 Β±0.018 |
| LightGBM | 0.9386 Β±0.0018 | 0.131 Β±0.018 | 0.9563 Β±0.0020 | 0.075 Β±0.024 |
| Gradient Boosting | 0.9383 Β±0.0013 | 0.186 Β±0.025 | 0.9566 Β±0.0011 | 0.106 Β±0.023 |
At F1.5, Random Forest edges out CatBoost by 0.0006 β well inside both models' std, i.e. not a meaningful difference. At F2, CatBoost and Random Forest are effectively tied (0.9580 vs 0.9579).
A plain-language translation of the metrics above, aimed at bedside decision-making rather than machine-learning benchmarking.
Logistic Regression calculates a weighted risk score from ~5 key lab values β similar in spirit to a clinical scoring system like BISAP, but the weights are learned directly from patient outcomes rather than expert consensus.
Unlike the Random Forest / Gradient Boosting / deep-learning models, its coefficients are fully transparent: you can see exactly which lab value pushed the prediction up or down for any individual patient.
β Lactate β β SAP risk (tissue hypoperfusion β consistent with known severity markers)
β Calcium β β SAP risk (established marker of fat necrosis / severity, mirrors Ranson criteria)
β Creatinine β β SAP risk (renal involvement, a recognized organ-failure marker)
AUC 0.817 β correctly ranks a random severe case above a random mild case ~82% of the time.
Sensitivity 93.8% β catches ~94 of 100 truly severe cases; few are missed.
Threshold 0.575 β the cutoff was chosen to favor not missing severe cases over avoiding false alarms.