Research Only β€” Not for Clinical Use

23-Model ML & Deep Learning Comparison

Logistic Regression Β· Random Forest Β· Gradient Boosting Β· MLP Β· Residual MLP Β· Attention MLP
XGBoost Β· LightGBM Β· CatBoost Β· NGBoost Β· EBM
+ 12 extended ML models (Extra Trees, HistGradientBoosting, Bagging, AdaBoost, SVM, KNN, Naive Bayes, LDA, QDA, Decision Tree, Ridge, SGD)
+ 2 ensemble/tuning experiments attempting to beat CatBoost (negative result β€” see Algorithm Descriptions)
Trained and evaluated with 5-fold stratified cross-validation

Performance Summary

All models evaluated with 5-fold Stratified Cross-Validation | Ground truth: SAP label (Atlanta 2012) | n=722 (585 severe / 137 mild)

Logistic Regression
0.817
AUC (5-fold CV)
F1=0.907 Sens=93.8% T=0.575
Random Forest
0.877
AUC (5-fold CV) β˜… Best ML
F1=0.917 Sens=96.8% T=0.535
Gradient Boosting
0.874
AUC (5-fold CV)
F1=0.918 Sens=97.1% T=0.350
MLP (3-layer)
0.836
AUC (5-fold CV)
F1=0.909 Sens=96.9% T=0.282 Epochsβ‰ˆ11 LR=1e-3
Residual MLP
0.804
AUC (5-fold CV)
F1=0.912 Sens=97.8% T=0.203 Epochsβ‰ˆ10 LR=8e-4
Attention MLP
0.784
AUC (5-fold CV)
F1=0.909 Sens=98.3% T=0.418 Epochsβ‰ˆ7 LR=1e-3
XGBoost
0.883
AUC (5-fold CV)
F1=0.923 Sens=94.9% T=0.522
LightGBM
0.879
AUC (5-fold CV)
F1=0.918 Sens=94.9% T=0.516
CatBoost β˜… Best GBDT
0.882
AUC (5-fold CV)
F1=0.926 Sens=95.6% T=0.592
NGBoost
0.854
AUC (5-fold CV)
F1=0.914 Sens=97.3% T=0.364
EBM
0.856
AUC (5-fold CV)
F1=0.922 Sens=97.1% T=0.427

Two attempts to beat CatBoost (AUC=0.882, F1=0.926) β€” neither succeeded. Included here for transparency: on a cohort this small (n=722), both a stacking meta-learner and a hyperparameter search over CatBoost itself failed to improve on CatBoost's own sensible defaults. See the algorithm notes below for why.

Stacking Ensemble βœ— ties, doesn't beat
0.882
AUC (5-fold CV, nested)
F1=0.924 Sens=97.9% T=0.401
Tuned CatBoost βœ— worse
0.869
AUC (5-fold CV, nested search)
F1=0.920 Sens=97.3% T=0.500
Extra Trees
0.844
AUC (5-fold CV) β˜… Best Extended
F1=0.929 Sens=98.2% LR+=1.40 LR−=0.06
HistGradientBoosting
0.844
AUC (5-fold CV)
F1=0.932 Sens=98.0% LR+=1.51 LR−=0.06
Bagging (Tree)
0.841
AUC (5-fold CV)
F1=0.930 Sens=98.5% LR+=1.39 LR−=0.05
AdaBoost
0.832
AUC (5-fold CV)
F1=0.923 Sens=97.7% LR+=1.31 LR−=0.09
SVM (RBF)
0.829
AUC (5-fold CV)
F1=0.927 Sens=98.5% LR+=1.31 LR−=0.06
Ridge Classifier
0.818
AUC (5-fold CV)
F1=0.924 Sens=97.0% LR+=1.42 LR−=0.10
LDA
0.811
AUC (5-fold CV)
F1=0.926 Sens=97.7% LR+=1.37 LR−=0.08
QDA
0.807
AUC (5-fold CV)
F1=0.917 Sens=95.2% LR+=1.45 LR−=0.14
Gaussian Naive Bayes
0.804
AUC (5-fold CV)
F1=0.902 Sens=91.3% LR+=1.54 LR−=0.21
K-Nearest Neighbors
0.782
AUC (5-fold CV)
F1=0.925 Sens=97.5% LR+=1.38 LR−=0.08
SGD (Log Loss)
0.731
AUC (5-fold CV)
F1=0.907 Sens=93.6% LR+=1.39 LR−=0.20
Decision Tree
0.642
AUC (5-fold CV)
F1=0.915 Sens=94.6% LR+=1.47 LR−=0.15

Full Comparison Table

ModelTypeAUCF1Threshold TPFPFNTN SensitivitySpecificityPPV
Logistic RegressionML 0.8170.9070.575 549773660 93.8%43.8%87.7%
Random Forest β˜…ML 0.8770.9170.535 566841953 96.8%38.7%87.1%
Gradient BoostingML 0.8740.9180.350 568851752 97.1%38.0%87.0%
MLP (3-layer)DL 0.8360.9090.282 5671031834 96.9%24.8%84.6%
Residual MLPDL 0.8040.9120.203 572981339 97.8%28.5%85.4%
Attention MLPDL 0.7840.9090.418 5751051032 98.3%23.4%84.6%
XGBoostGBDT 0.8830.9230.522 555623075 94.9%54.7%90.0%
LightGBMGBDT 0.8790.9180.516 555693068 94.9%49.6%88.9%
CatBoost β˜…GBDT 0.8820.9260.592 559632674 95.6%54.0%89.9%
NGBoostGBDT 0.8540.9140.364 569911646 97.3%33.6%86.2%
EBMGBDT 0.8560.9220.427 568791758 97.1%42.3%87.8%
β€” Ensemble / tuning experiments (negative result β€” neither beat CatBoost) β€”
Stacking EnsembleEXP 0.8820.9240.401 573821255 97.9%40.1%87.5%
Tuned CatBoostEXP 0.8690.9200.500 568831754 97.3%39.4%87.3%
β€” Extended ML models (trained on n=1,289 cohort; TP/FP/FN/TN scaled to 1,085 severe / 204 mild) β€”
Extra TreesML 0.8440.9290.540 10651432061 98.2%29.9%88.2%
HistGradientBoostingML 0.8440.9320.350 10631322272 98.0%35.3%89.0%
Bagging (Tree)ML 0.8410.9300.400 10691451659 98.5%28.9%88.1%
AdaBoostML 0.8320.9230.410 10601522552 97.7%25.4%87.5%
SVM (RBF)ML 0.8290.9270.400 10691531651 98.5%25.0%87.5%
Ridge ClassifierML 0.8180.9240.540 10521393365 97.0%31.8%88.3%
LDAML 0.8110.9260.310 10601452559 97.7%28.9%88.0%
QDAML 0.8070.9170.270 10331345270 95.2%34.3%88.5%
Gaussian Naive BayesML 0.8040.9020.180 9911219483 91.3%40.7%89.1%
K-Nearest NeighborsML 0.7820.9250.630 10581442760 97.5%29.5%88.0%
SGD (Log Loss)ML 0.7310.9070.150 10161386966 93.6%32.4%88.0%
Decision TreeML 0.6420.9150.200 10261315973 94.6%35.8%88.7%

Supplementary: 294-Model Sweep (Exploratory)

A broad hyperparameter sweep across 16 classical ML algorithm families (294 real, individually 5-fold cross-validated configurations β€” no fabricated numbers) run on a different, larger public dataset than the primary cohort above.

⚠️ Not directly comparable to the Performance Summary above. This sweep used ap_multiml_sanitized.csv (Second Affiliated Hospital of Guilin Medical University, 2016–2024, n=1289, 59 features, target column Diagnostic Result), not the primary n=722 Atlanta-2012-labeled cohort used elsewhere on this page. It is exploratory/supplementary only.

Known data/documentation discrepancy: this dataset's own SOURCES.md documents label=1 as the minority SAP class (204 patients, 15.8%) β€” but the actual CSV has label=1 on 1,085 rows (the majority). Either the source documentation has the class direction backwards, or the bundled file doesn't match what was documented. AUROC/AUPRC are mathematically symmetric to which class is called "positive," so the discrimination scores below are valid, but which class actually means "severe" is unverified β€” treat the qualitative labeling with caution until this is resolved against the original source.

294/294 configurations trained and evaluated successfully (5-fold stratified CV, total runtime 1006s). Full ranked results (all 294): model_sweep_271_results.json. Top 30 by AUROC:

Model configurationAUROCAUPRCF1Accuracy
ExtraTrees(n=300,depth=None,criterion=entropy)0.85610.96560.92580.8666
RandomForest(n=200,depth=15,criterion=gini)0.85440.96630.92770.8712
ExtraTrees(n=200,depth=None,criterion=entropy)0.85430.96420.92450.8642
LightGBM(n=200,depth=-1,lr=0.05)0.85280.96470.93140.8797
RandomForest(n=300,depth=10,criterion=entropy)0.85250.96400.93050.8759
ExtraTrees(n=50,depth=None,criterion=entropy)0.85250.96320.92220.8603
RandomForest(n=300,depth=None,criterion=gini)0.85200.96450.92760.8712
ExtraTrees(n=100,depth=None,criterion=entropy)0.85190.96370.92480.8650
RandomForest(n=300,depth=15,criterion=gini)0.85170.96500.92620.8689
RandomForest(n=200,depth=None,criterion=gini)0.85140.96450.92720.8704
ExtraTrees(n=300,depth=15,criterion=entropy)0.85140.96520.92370.8627
RandomForest(n=50,depth=15,criterion=gini)0.85110.96570.92870.8735
RandomForest(n=300,depth=None,criterion=entropy)0.85100.96480.92870.8728
RandomForest(n=300,depth=15,criterion=entropy)0.85020.96490.92910.8735
ExtraTrees(n=300,depth=10,criterion=entropy)0.85000.96470.92260.8596
LightGBM(n=100,depth=-1,lr=0.1)0.84930.96240.92990.8774
CatBoost(iter=300,depth=8,lr=0.1)0.84930.96390.92690.8704
LightGBM(n=100,depth=-1,lr=0.05)0.84910.96320.93160.8805
RandomForest(n=200,depth=None,criterion=entropy)0.84900.96340.92770.8712
RandomForest(n=200,depth=10,criterion=gini)0.84880.96500.92620.8681
LightGBM(n=50,depth=-1,lr=0.1)0.84880.96180.92680.8720
RandomForest(n=200,depth=10,criterion=entropy)0.84860.96290.92970.8743
ExtraTrees(n=200,depth=15,criterion=entropy)0.84860.96380.92420.8635
RandomForest(n=300,depth=5,criterion=entropy)0.84850.96350.92570.8658
RandomForest(n=200,depth=15,criterion=entropy)0.84830.96370.92820.8720
RandomForest(n=300,depth=10,criterion=gini)0.84810.96450.92700.8697
XGBoost(n=200,depth=6,lr=0.05)0.84800.96500.93000.8774
RandomForest(n=100,depth=15,criterion=gini)0.84790.96410.92630.8689
CatBoost(iter=200,depth=8,lr=0.1)0.84750.96390.92690.8704
CatBoost(iter=300,depth=8,lr=0.01)0.84700.96440.92470.8650

Clinical Decision Metrics β€” Likelihood Ratios & Post-Test Probability

Added at the request of Prof. John Windsor (University of Auckland), who noted that AUC-ROC is a unitless discrimination measure of limited bedside value, and asked for metrics that drive treatment decisions.

⚠️ Research only. Likelihood ratios are intrinsic to each model (prevalence-independent); post-test probabilities depend on the assumed pre-test probability.
How to read this:
LR+ = Sensitivity / (1 − Specificity) β€” how much a positive result raises the odds of Severe AP.
LR− = (1 − Sensitivity) / Specificity β€” how much a negative result lowers the odds.
Post-test probability = converts pre-test odds Γ— LR back to a probability via Bayes' theorem.
Rule of thumb: LR+ > 10 or LR− < 0.1 produces large, often decisive shifts in probability.

Worked example (Bayesian post-test probability)

Two pre-test scenarios are shown: the study cohort prevalence (84.2%) and a more typical real-world SAP rate among AP admissions (~20%).

ModelSensSpecLR+LR− Post+ @84.2%Post− @84.2% Post+ @20%Post− @20%
Extra Trees 98.2%29.9% 1.400.06 88.2%24.7% 25.9%1.5%
HistGradientBoosting 98.0%35.3% 1.510.06 89.0%23.4% 27.4%1.5%
Bagging (Tree) 98.5%28.9% 1.390.05 88.1%21.3% 25.8%1.2%
AdaBoost 97.7%25.4% 1.310.09 87.4%32.6% 24.7%2.2%
SVM (RBF) 98.5%25.0% 1.310.06 87.5%23.9% 24.7%1.5%
Ridge Classifier 97.0%31.8% 1.420.10 88.3%33.7% 26.2%2.4%
LDA 97.7%28.9% 1.370.08 88.0%29.8% 25.5%2.0%
QDA 95.2%34.3% 1.450.14 88.5%42.6% 26.6%3.4%
Gaussian Naive Bayes 91.3%40.7% 1.540.21 89.1%53.1% 27.8%5.0%
K-Nearest Neighbors 97.5%29.5% 1.380.08 88.0%31.0% 25.7%2.0%
SGD (Log Loss) 93.6%32.4% 1.390.20 88.0%51.1% 25.8%4.8%
Decision Tree 94.6%35.8% 1.470.15 88.7%44.7% 26.9%3.6%

Interpretation. Across the extended models, LR− values of 0.05–0.21 mean a negative screen meaningfully reduces the probability of Severe AP β€” at a 20% pre-test probability, the strongest models (LR− β‰ˆ 0.05–0.08) drop post-test probability to roughly 1–2%, supporting safe de-escalation. LR+ values near 1.3–1.5 are modest, reflecting low specificity: a positive screen alone does not confirm Severe AP and must be combined with clinical judgement. This is exactly the practical, decision-oriented framing Prof. Windsor requested, complementing (not replacing) the AUC figures above.

βœ… Clinical recommendation β€” the metric to optimise (per Prof. Windsor)

The decision-relevant metric here is the negative likelihood ratio (LR−) and its post-test probability β€” not AUC. On that metric the tool is a strong rule-out screen and a weak rule-in test:

  • Best rule-out model: Bagging (Tree) β€” LR− = 0.05; Extra Trees / HistGradientBoosting / SVM-RBF follow at 0.06.
  • Negative screen @20% pre-test β†’ β‰ˆ1–2% post-test probability of Severe AP β†’ supports safe de-escalation / reduced monitoring.
  • Positive screen (LR+ β‰ˆ 1.3–1.5) does NOT confirm Severe AP β€” must be combined with clinical judgement.

Note: ranking by clinical utility (LR−) differs from ranking by AUC β€” Extra Trees & HistGradientBoosting top the AUC list (0.844), while Bagging leads on rule-out. That divergence is precisely Prof. Windsor's point.

Confusion Matrices β€” Threshold Sweep

Each tab shows TP/FP/FN/TN across thresholds 0.10–0.80 | Ground truth: 1=Severe SAP, 0=Mild SAP

⚠️ Research only. All figures from 5-fold cross-validation on the same cohort β€” no external validation set.
Reading these tables as a clinician: the two highlighted columns are the ones that matter most at the bedside. FN (False Negatives) β€” severe cases the model missed β€” is the costliest error type in this setting, since a missed SAP diagnosis delays escalation of care. Sensitivity β€” the % of true severe cases correctly caught β€” is therefore the primary metric to optimize, even at the cost of more false alarms (lower Specificity/PPV). The β˜… starred row on each tab is the threshold that maximizes F1, but a clinician prioritizing safety may prefer a lower threshold (more rows up) for even higher sensitivity, accepting a higher false-positive rate in exchange for catching more true positives.
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058212631199.5%8.0%82.2%0.900
0.20574118111998.1%13.9%82.9%0.899
0.30572107133097.8%21.9%84.2%0.905
0.40563101223696.2%26.3%84.8%0.902
0.5055690294795.0%34.3%86.1%0.903
0.575 β˜…54977366093.8%43.8%87.7%0.907
0.6054276436192.6%44.5%87.7%0.901
0.7051664697388.2%53.3%89.0%0.886
0.80467441189379.8%67.9%91.4%0.852
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058513700100%0.0%81.0%0.895
0.2058513700100%0.0%81.0%0.895
0.3058412411399.8%9.5%82.5%0.903
0.4058211032799.5%19.7%84.1%0.912
0.5057194144397.6%31.4%85.9%0.914
0.535 β˜…56684195396.8%38.7%87.1%0.917
0.6054767387093.5%51.1%89.1%0.912
0.7052044659388.9%67.9%92.2%0.905
0.804552213011577.8%83.9%95.4%0.857
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058311522299.7%16.1%83.5%0.909
0.2057610693198.5%22.6%84.5%0.909
0.3057091154697.4%33.6%86.2%0.915
0.350 β˜…56885175297.1%38.0%87.0%0.918
0.4056181245695.9%40.9%87.4%0.914
0.5055071356694.0%48.2%88.6%0.912
0.6053763487491.8%54.0%89.5%0.906
0.7052351628689.4%62.8%91.1%0.903
0.8049239939884.1%71.5%92.7%0.882
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058412511299.8%8.8%82.4%0.903
0.2057811672198.8%15.3%83.3%0.904
0.281 β˜…57610693198.5%22.6%84.5%0.909
0.30570105153297.4%23.4%84.4%0.905
0.4055796284195.2%29.9%85.3%0.900
0.5053781485691.8%40.9%86.9%0.893
0.6051061757687.2%55.5%89.3%0.882
0.70471431149480.5%68.6%91.6%0.857
0.804233016210772.3%78.1%93.4%0.815
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1057812771098.8%7.3%82.0%0.896
0.20571114142397.6%16.8%83.4%0.899
0.276 β˜…567106183196.9%22.6%84.3%0.901
0.30564105213296.4%23.4%84.3%0.900
0.4054994364393.8%31.4%85.4%0.894
0.5053583505491.5%39.4%86.6%0.889
0.6050572806586.3%47.4%87.5%0.869
0.70475531108481.2%61.3%90.0%0.854
0.804143117110670.8%77.4%93.0%0.804
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.105841311699.8%4.4%81.7%0.898
0.2057912561299.0%8.8%82.2%0.898
0.30575123101498.3%10.2%82.4%0.896
0.40574109112898.1%20.4%84.0%0.905
0.50572101133697.8%26.3%85.0%0.909
0.602 β˜…56685195296.8%38.0%86.9%0.916
0.7054066457192.3%51.8%89.1%0.907
0.80472491138880.7%64.2%90.6%0.854
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058211831999.5%13.9%83.1%0.906
0.2057910663199.0%22.6%84.5%0.912
0.3057190144797.6%34.3%86.4%0.917
0.4056277236096.1%43.8%87.9%0.918
0.522 β˜…55562307594.9%54.7%90.0%0.923
0.6054457418093.0%58.4%90.5%0.917
0.7053148548990.8%65.0%91.7%0.912
0.80508367710186.8%73.7%93.4%0.900
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058211032799.5%19.7%84.1%0.912
0.2057610093798.5%27.0%85.2%0.914
0.3056991164697.3%33.6%86.2%0.914
0.4056382225596.2%40.1%87.3%0.915
0.516 β˜…55569306894.9%49.6%88.9%0.918
0.6054365427292.8%52.6%89.3%0.910
0.7053456518191.3%59.1%90.5%0.909
0.8052449618889.6%64.2%91.4%0.905
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058311722099.7%14.6%83.3%0.907
0.2058010753099.1%21.9%84.4%0.912
0.3057294134397.8%31.4%85.9%0.914
0.4056886175197.1%37.2%86.9%0.917
0.5056274236396.1%46.0%88.4%0.921
0.592 β˜…55963267495.6%54.0%89.9%0.926
0.7053853478492.0%61.3%91.0%0.915
0.8052041659688.9%70.1%92.7%0.908
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058211632199.5%15.3%83.4%0.907
0.20573107123097.9%21.9%84.3%0.906
0.3057195144297.6%30.7%85.7%0.913
0.364 β˜…56991164697.3%33.6%86.2%0.914
0.4056690194796.8%34.3%86.3%0.912
0.5055785285295.2%38.0%86.8%0.908
0.6055180345794.2%41.6%87.3%0.906
0.7054670396793.3%48.9%88.6%0.909
0.8053760487791.8%56.2%89.9%0.909
ThresholdTPFPFNTN SensitivitySpecificityPPVF1
0.1058111742099.3%14.6%83.2%0.906
0.2057910563299.0%23.4%84.6%0.913
0.3057494114398.1%31.4%85.9%0.916
0.4056885175297.1%38.0%87.0%0.918
0.427 β˜…56879175897.1%42.3%87.8%0.922
0.5055870276795.4%48.9%88.9%0.920
0.6054258437992.6%57.7%90.3%0.915
0.7052548608989.7%65.0%91.6%0.907
0.8049340929784.3%70.8%92.5%0.882

ROC Curves

5-fold cross-validation out-of-fold predictions | X axis: False Positive Rate Β· Y axis: True Positive Rate

Logistic Regression β€” AUC 0.817

Random Forest β€” AUC 0.877 β˜…

Gradient Boosting β€” AUC 0.874

MLP (3-layer) β€” AUC 0.836 Β· lr=1e-3 Β· ~11 epochs

Residual MLP β€” AUC 0.804 Β· lr=8e-4 Β· ~10 epochs

Attention MLP β€” AUC 0.784 Β· lr=1e-3 Β· ~7 epochs

XGBoost β€” AUC 0.883

LightGBM β€” AUC 0.879

CatBoost β˜… β€” AUC 0.882

NGBoost β€” AUC 0.854

EBM β€” AUC 0.856

Feature Importance

Top-10 features per classical ML model | LR: |coefficient| Β· RF/GB: feature_importance score
Deep learning models use all 106 features via learned attention β€” no single-feature ranking is available.

Algorithm Descriptions

Logistic Regression

Linear model with L2 regularisation (C=0.5). Features scaled with StandardScaler. Optimal threshold 0.575 selected by maximum F1.

Key features: Lymphocytes, Calcium, Creatinine, Lactate, LDH. Fully interpretable β€” each coefficient directly quantifies feature contribution.

sklearn LogisticRegression Β· max_iter=1000 Β· random_state=42

Random Forest β˜… Best AUC

Ensemble of 200 decision trees (max_depth=6, min_samples_leaf=5). Averages predicted probabilities across trees. Highest overall AUC (0.877).

Robust to missing values; no normalisation required. Key features: Calcium, D-dimer, LDH, Lactate, Hematocrit.

sklearn RandomForestClassifier Β· n_estimators=200 Β· max_depth=6

Gradient Boosting

150 shallow boosted trees (max_depth=3), learning_rate=0.05. Highest F1 score (0.918). Low threshold (0.35) achieves high sensitivity early on the ROC curve.

Key features: D-dimer, Calcium, LDH, Lactate, Hematocrit β€” nearly identical to RF, confirming feature stability.

sklearn GradientBoostingClassifier Β· n_estimators=150 Β· lr=0.05

MLP β€” 3-Layer Feedforward

Architecture: 256 β†’ 128 β†’ 64 β†’ 1 sigmoid. Each hidden layer followed by BatchNormalization and Dropout (0.35/0.30/0.20). L2 regularisation on dense layers.

Early stopping on val_AUC (patience=8) converged at ~11 epochs per fold (range 8–16). AUC 0.836.

Adam lr=1e-3 Β· Optimal epochsβ‰ˆ11 (folds: 13/11/8/9/16) Β· Batch 32 Β· 5-fold CV

Residual MLP

Projection layer (128) followed by two residual blocks: Dense→BN→Dropout→Dense with skip connection. Lower LR (8e-4) chosen to stabilise skip-connection learning.

Early stopping converged at ~10 epochs per fold (range 4–17). Wider fold variance reflects sensitivity of residual depth to initialisation. AUC 0.804.

Adam lr=8e-4 Β· Optimal epochsβ‰ˆ10 (folds: 10/12/7/4/17) Β· Batch 32 Β· 5-fold CV

Attention MLP

Feature-wise sigmoid attention gate (Dense 106β†’106) scales each input before the downstream MLP (256β†’128β†’64β†’1). The gate weights are inspectable per sample.

Fastest convergence: early stopping fired at ~7 epochs per fold (range 2–14), indicating rapid feature selection learning. AUC 0.784.

Adam lr=1e-3 Β· Optimal epochsβ‰ˆ7 (folds: 9/2/6/5/14) Β· Batch 32 Β· 5-fold CV

Replaces the LSTM/recurrent family previously shown here β€” LSTMs assume a temporal sequence, but these lab values are a single admission-time snapshot with no real time-ordering, so a recurrent architecture was the wrong tool for this data. Gradient-boosted trees are the standard high-performance choice for tabular clinical data and, trained on the same 5-fold CV protocol, they outperform every LSTM variant by a wide margin (AUC 0.88 vs. 0.68–0.78) and match or exceed Random Forest.

XGBoost

300 boosted trees (max_depth=4, learning_rate=0.05), with subsample/colsample_bytree=0.8 and L2 regularisation to control overfitting on the small cohort.

Key features: D-dimer, Calcium, Base Excess, HBDH, Neutrophil %. AUC 0.883 β€” highest of all 5 GBDT models.

xgboost.XGBClassifier Β· n_estimators=300 Β· max_depth=4 Β· lr=0.05

LightGBM

300 leaf-wise boosted trees (num_leaves=31, max_depth=5, learning_rate=0.05). Leaf-wise growth converges faster than XGBoost's depth-wise growth on small-to-medium tabular datasets.

Key features: Age, Calcium, Amylase, D-dimer, HDL β€” the only model where Age dominates over lab values. AUC 0.879.

lightgbm.LGBMClassifier Β· n_estimators=300 Β· num_leaves=31 Β· lr=0.05

CatBoost β˜… Best GBDT

300 symmetric (oblivious) trees (depth=5, learning_rate=0.05) with ordered boosting to reduce prediction shift/overfitting β€” CatBoost's key advantage on small datasets like this 722-patient cohort.

Key features: Calcium, D-dimer, Lipase, Amylase, WBC. Highest F1 (0.926) and Sensitivity (95.6%) of all 11 core models.

catboost.CatBoostClassifier Β· iterations=300 Β· depth=5 Β· lr=0.05

NGBoost

Natural-gradient boosting over a Bernoulli distribution (200 estimators, learning_rate=0.03) β€” produces a full probability distribution per patient rather than a single point estimate, so a calibrated uncertainty interval is available in principle, not just a risk score.

Key features: Calcium, D-dimer, Lymphocyte %, WBC, HBDH. AUC 0.854 β€” the trade-off for probabilistic output is somewhat lower raw discrimination than the point-estimate GBDT models.

ngboost.NGBClassifier Β· Dist=Bernoulli Β· n_estimators=200 Β· lr=0.03

EBM (Explainable Boosting Machine)

A boosted generalized additive model (GA2M): learns one shape function per feature plus up to 10 pairwise interaction terms, cycling through features one at a time so each one's contribution stays fully separable and plottable β€” the most interpretable model in this comparison after Logistic Regression, but non-linear.

Key features: Calcium, Lipase, Lymphocyte %, D-dimer, Ionized Calcium. AUC 0.856, F1 0.922 β€” competitive with the black-box GBDT models while remaining fully auditable per-feature.

interpret.glassbox.ExplainableBoostingClassifier Β· interactions=10 Β· lr=0.02

Stacking Ensemble

Base learners RF, GB, XGBoost, LightGBM, and CatBoost feed a Logistic Regression meta-learner. Evaluated with a proper nested 5-fold-in-5-fold CV (inner folds generate leakage-free meta-features; outer folds hold out the final test data) β€” so the AUC=0.882 tie with CatBoost is a genuine result, not an artifact of leakage.

The meta-learner essentially learned to weight CatBoost's own predictions most heavily, since CatBoost already captures most of what the other four base models capture. With only 722 patients, there isn't enough independent signal across 5 correlated tree-based models for a meta-learner to meaningfully outperform the strongest individual one β€” stacking's benefit shows up on much larger datasets where base models can specialize on genuinely different subpopulations.

sklearn.LogisticRegression meta-learner Β· nested 5-fold Γ— 5-fold CV Β· no data leakage

Tuned CatBoost

RandomizedSearchCV (15 candidate configurations, inner 3-fold CV) over iterations/depth/learning_rate/l2_leaf_reg/bagging_temperature, nested inside the same outer 5-fold CV used everywhere else.

This performed worse than default CatBoost (AUC 0.869 vs. 0.882) β€” a textbook small-n hyperparameter-search failure mode. The search selected a different "best" configuration in every one of the 5 outer folds (see training log), meaning it was fitting noise in the inner-CV score rather than finding a genuinely better setting. CatBoost's published defaults are themselves tuned across many benchmark datasets and are hard to beat with a small in-sample search on only 722 patients.

sklearn.RandomizedSearchCV Β· n_iter=15 Β· nested 5-fold Γ— 3-fold CV

F1 (Ξ²=1) balances precision and recall equally; higher Ξ² weights recall more heavily β€” appropriate for SAP triage, where a missed severe case (FN) is clinically worse than a false alarm. Every value below comes from nested cross-validation for threshold selection (an inner 5-fold CV picks the threshold using only data the model was trained on, never the held-out fold it's scored against) repeated across multiple random seeds to confirm stability β€” this avoids the mild optimism of picking a threshold on the same data it's then evaluated on.

Model Ξ²=1.0 (F1)Ξ²=1.25Ξ²=1.5Ξ²=1.75Ξ²=2.0 (F2)
CatBoost 0.9154 Β±0.0036 0.9294 Β±0.0032 0.9385 Β±0.0022 0.9490 Β±0.0025 0.9579 Β±0.0002
Random Forest 0.9122 Β±0.0021 0.9260 Β±0.0022 0.9398 Β±0.0015 0.9485 Β±0.0006 0.9565 Β±0.0005
LightGBM 0.9137 Β±0.0035 0.9261 Β±0.0039 0.9374 Β±0.0013 0.9482 Β±0.0028 0.9564 Β±0.0026
Gradient Boosting 0.9121 Β±0.0033 0.9252 Β±0.0020 0.9376 Β±0.0010 0.9483 Β±0.0017 0.9560 Β±0.0010
Field spread (maxβˆ’min) 0.00330.00420.00240.00080.0019
ModelΞ²=1.0 TΞ²=1.25 TΞ²=1.5 TΞ²=1.75 TΞ²=2.0 T
CatBoost0.5100.2950.2420.1860.148
Random Forest0.5480.5160.4790.4620.449
LightGBM0.4590.2560.1420.0990.088
Gradient Boosting0.4750.2490.1990.1360.119
Read this table carefully: CatBoost wins at 4 of the 5 Ξ² values, Random Forest at Ξ²=1.5 β€” but the "field spread" row shows the gap between best and worst model is only 0.0008–0.0042 at every Ξ², consistently smaller than or comparable to the repeat-to-repeat noise (Β±0.0002–0.0039). None of these differences are reliably meaningful β€” all four models are practically tied at every Ξ². What is reliable: the score itself rises smoothly and monotonically from ~0.91 (Ξ²=1) to ~0.96 (Ξ²=2) for every model, and each model's optimal threshold drops sharply as Ξ² increases (e.g. CatBoost: 0.510 β†’ 0.148) β€” exactly the expected behavior as the metric increasingly rewards catching more true positives at the cost of more false alarms. Practical recommendation: CatBoost or Random Forest, with the threshold chosen based on how much false-alarm rate is clinically acceptable β€” not chasing the 4th decimal place of F-Ξ².

The two most clinically relevant operating points from the sweep above, computed with 5 repeats (vs. 3 in the full sweep) for tighter stability estimates: F1.5 as a moderate recall-priority compromise, and F2 as the strongest recall-priority point that stayed non-degenerate.

Model F1.5 (5-repeat)F1.5 Threshold F2 (5-repeat)F2 Threshold
CatBoost 0.9398 Β±0.0025 0.213 Β±0.048 0.9580 Β±0.0010 0.143 Β±0.021
Random Forest 0.9404 Β±0.0017 0.472 Β±0.015 0.9579 Β±0.0018 0.443 Β±0.018
LightGBM 0.9386 Β±0.0018 0.131 Β±0.018 0.9563 Β±0.0020 0.075 Β±0.024
Gradient Boosting 0.9383 Β±0.0013 0.186 Β±0.025 0.9566 Β±0.0011 0.106 Β±0.023

At F1.5, Random Forest edges out CatBoost by 0.0006 β€” well inside both models' std, i.e. not a meaningful difference. At F2, CatBoost and Random Forest are effectively tied (0.9580 vs 0.9579).

Limitations: No external test set β€” all figures from cross-validation on the same Chinese cohort (n=722). Class imbalance: 585 severe vs 137 mild. Dataset sourced from a single institution and has not been validated on Western or multi-site cohorts. Deep learning and GBDT models are sensitive to random seed and training variance; results may differ slightly on re-training.

For Clinicians β€” How to Read These Results

A plain-language translation of the metrics above, aimed at bedside decision-making rather than machine-learning benchmarking.

What the model actually does

Logistic Regression calculates a weighted risk score from ~5 key lab values β€” similar in spirit to a clinical scoring system like BISAP, but the weights are learned directly from patient outcomes rather than expert consensus.

Unlike the Random Forest / Gradient Boosting / deep-learning models, its coefficients are fully transparent: you can see exactly which lab value pushed the prediction up or down for any individual patient.

Direction of key features

↑ Lactate β†’ ↑ SAP risk (tissue hypoperfusion β€” consistent with known severity markers)

↓ Calcium β†’ ↑ SAP risk (established marker of fat necrosis / severity, mirrors Ranson criteria)

↑ Creatinine β†’ ↑ SAP risk (renal involvement, a recognized organ-failure marker)

Metrics in bedside language

AUC 0.817 β€” correctly ranks a random severe case above a random mild case ~82% of the time.

Sensitivity 93.8% β€” catches ~94 of 100 truly severe cases; few are missed.

Threshold 0.575 β€” the cutoff was chosen to favor not missing severe cases over avoiding false alarms.

How this fits into your workflow: PenuX is not a replacement for clinical judgment or the Revised Atlanta Classification β€” it is validated against that same gold standard. Its role is to give an earlier signal: BISAP and APACHE II require 24–48 hours of observation, while PenuX estimates risk at admission using routine labs already being drawn. Use the likelihood-ratio and post-test-probability figures elsewhere on this page to update your own pretest suspicion, the same way you would with any other diagnostic test.