Research article Special Issues

A leakage-free and risk-aware machine learning framework for reliable diabetes prediction

  • Published: 23 October 2026
  • Reliable diabetes prediction requires more than high discrimination; clinical risk models should also provide reliable probability estimates and account for the unequal consequences of prediction errors. This study proposes a leakage-free and risk-aware machine learning framework that integrates training-only preprocessing, probability calibration, cost-sensitive thresholding, statistical uncertainty assessment, subgroup stability analysis, and decision curve analysis (DCA). The framework was independently evaluated on the Pima Indians Diabetes Dataset (PIMA; N = 768) and the CDC Diabetes Health Indicators dataset (N = 253,680) using outer cross-validation, with preprocessing, model fitting, calibration, and threshold tuning performed without using the corresponding outer test folds. Logistic regression achieved the highest ROC-AUC on PIMA (0.8431; 95% CI: 0.8151–0.8695), while gradient boosting achieved the highest ROC-AUC on the CDC dataset (0.8298; 95% CI: 0.8276–0.8319). The calibration performance varied between methods and metrics, and there was no single method that consistently outperformed the others. Cost-sensitive analysis showed that lowering the decision threshold under higher assumed false-negative costs increased recall and reduced missed positive cases, but at the expense of more false positives. This trade-off was consistent across cost-ratio sensitivity analyses, while DCA indicated a positive model net benefit around the decision threshold corresponding to the primary 5:1 cost scenario in both datasets. In general, the results indicate that reliable evaluation of the diabetes risk model should extend beyond discrimination alone and jointly consider leakage control, probability calibration, asymmetric error costs, statistical uncertainty, subgroup stability, and potential decision utility.

    Citation: Didar Abdalrahman Ismael, Aram Rasul. A leakage-free and risk-aware machine learning framework for reliable diabetes prediction[J]. AIMS Bioengineering, 2026, 13(4): 483-503. doi: 10.3934/bioeng.2026023

    Related Papers:

  • Reliable diabetes prediction requires more than high discrimination; clinical risk models should also provide reliable probability estimates and account for the unequal consequences of prediction errors. This study proposes a leakage-free and risk-aware machine learning framework that integrates training-only preprocessing, probability calibration, cost-sensitive thresholding, statistical uncertainty assessment, subgroup stability analysis, and decision curve analysis (DCA). The framework was independently evaluated on the Pima Indians Diabetes Dataset (PIMA; N = 768) and the CDC Diabetes Health Indicators dataset (N = 253,680) using outer cross-validation, with preprocessing, model fitting, calibration, and threshold tuning performed without using the corresponding outer test folds. Logistic regression achieved the highest ROC-AUC on PIMA (0.8431; 95% CI: 0.8151–0.8695), while gradient boosting achieved the highest ROC-AUC on the CDC dataset (0.8298; 95% CI: 0.8276–0.8319). The calibration performance varied between methods and metrics, and there was no single method that consistently outperformed the others. Cost-sensitive analysis showed that lowering the decision threshold under higher assumed false-negative costs increased recall and reduced missed positive cases, but at the expense of more false positives. This trade-off was consistent across cost-ratio sensitivity analyses, while DCA indicated a positive model net benefit around the decision threshold corresponding to the primary 5:1 cost scenario in both datasets. In general, the results indicate that reliable evaluation of the diabetes risk model should extend beyond discrimination alone and jointly consider leakage control, probability calibration, asymmetric error costs, statistical uncertainty, subgroup stability, and potential decision utility.


    Abbreviations

    AUC

    area under the receiver operating characteristic curve

    BS

    Brier score

    CDC

    Centers for Disease Control and Prevention

    CI

    confidence interval

    DCA

    decision curve analysis

    ECE

    expected calibration error

    FN

    false negative

    FNR

    false-negative rate

    FP

    false positive

    FPR

    false-positive rate

    GB

    gradient boosting

    LR

    logistic regression

    MCE

    maximum calibration error

    ML

    machine learning

    PIMA

    Pima Indians Diabetes Dataset

    RF

    random forest

    ROC

    receiver operating characteristic

    SVM

    support vector machine.

    加载中


    Conflict of interest



    The authors declare that they have no conflicts of interest with respect to this paper.

    [1] Rajkomar A, Oren E, Chen K, et al. (2018) Scalable and accurate deep learning with electronic health records. npj Digit Med 1: 18. https://doi.org/10.1038/s41746-018-0029-1
    [2] Sisodia D, Sisodia DS (2018) Prediction of diabetes using classification algorithms. Procedia Comput Sci 132: 1578-1585. https://doi.org/10.1016/j.procs.2018.05.122
    [3] Kaufman S, Rosset S, Perlich C, et al. (2012) Leakage in data mining: formulation, detection, and avoidance. ACM Trans Knowl Discov Data 6: 1-21. https://doi.org/10.1145/2382577.2382579
    [4] Kapoor S, Narayanan A (2023) Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4: 100804. https://doi.org/10.1016/j.patter.2023.100804
    [5] Niculescu-Mizil A, Caruana R (2005) Predicting good probabilities with supervised learning. Proceedings of the International Conference on Machine Learning : 625-632. https://doi.org/10.1145/1102351.1102430
    [6] Van Calster B, McLernon DJ, van Smeden M, et al. (2019) Calibration: the Achilles heel of predictive analytics. BMC Med 17: 230. https://doi.org/10.1186/s12916-019-1466-7
    [7] Elkan C (2001) The foundations of cost-sensitive learning. Proceedings of the International Joint Conference on Artificial Intelligence : 973-978.
    [8] Provost F, Fawcett T (1997) Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. Proceedings of the Third International Conference on Knowledge Discovery and Data Mining : 43-48.
    [9] Vickers AJ, Elkin EB (2006) Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 26: 565-574. https://doi.org/10.1177/0272989X06295361
    [10] Vickers AJ, van Calster B, Steyerberg EW (2019) A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res 3: 18. https://doi.org/10.1186/s41512-019-0064-7
    [11] Kuhn M, Johnson K (2013) Applied Predictive Modeling. New York: Springer. https://doi.org/10.1007/978-1-4614-6849-3
    [12] Alturayeif N, Hassine J (2025) Data leakage detection in machine learning code: transfer learning, active learning, or low-shot prompting?. PeerJ Comput Sci 11: e2730. https://doi.org/10.7717/peerj-cs.2730
    [13] Rosenblatt M, Tejavibulya L, Jiang R, et al. (2024) Data leakage inflates prediction performance in connectome-based machine learning models. Nat Commun 15: 1829. https://doi.org/10.1038/s41467-024-46150-w
    [14] Amma BNG (2024) En-RfRsK: an ensemble machine learning technique for prognostication of diabetes mellitus. Egypt Inform J 25: 100441. https://doi.org/10.1016/j.eij.2024.100441
    [15] Kawarkhe M, Kaur P (2024) Prediction of diabetes using diverse ensemble learning classifiers. Procedia Comput Sci 235: 403-413. https://doi.org/10.1016/j.procs.2024.04.040
    [16] Jena SK, Behera DK, Jena AK, et al. (2025) A novel ensemble machine learning framework for improved diabetes prediction and complication prevention. Procedia Comput Sci 258: 4008-4017. https://doi.org/10.1016/j.procs.2025.04.652
    [17] Rizal, Azzahra AF, Bandaso RMS, et al. (2025) Type 2 diabetes prediction using recall-optimized XGBoost. Procedia Comput Sci 269: 1309-1318. https://doi.org/10.1016/j.procs.2025.09.072
    [18] Phan LT, Rakkiyappan R, Manavalan B (2025) ReMed-T2D: a robust ensemble learning model for early detection of type 2 diabetes using healthcare dataset. Comput Biol Med 187: 109771. https://doi.org/10.1016/j.compbiomed.2025.109771
    [19] Khan S, Naeem M, Qiyas M (2023) Deep intelligent predictive model for the identification of diabetes. AIMS Math 8: 16446-16462. https://doi.org/10.3934/math.2023840
    [20] Khan S (2026) Biochemical biomarker-driven deep learning framework with SHAP-based feature interpretation for diabetes classification. Biophys Chem 334: 107601. https://doi.org/10.1016/j.bpc.2026.107601
    [21] Almusallam N, Khan S, Alarfaj FK, et al. (2025) A robust deep learning framework for RNA 5-methyluridine modification prediction using integrated features. BMC Biol 23: 328. https://doi.org/10.1186/s12915-025-02433-2
    [22] Khushal R, Fatima U (2024) Fuzzy logic and machine learning for diabetes risk prediction using modifiable factors. Int J Adv Appl Sci 11: 225-231. https://doi.org/10.21833/ijaas.2024.12.025
    [23] Platt J (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers 61-74. https://doi.org/10.7551/mitpress/1113.003.0008
    [24] Zadrozny B, Elkan C (2002) Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining : 694-699. https://doi.org/10.1145/775047.775151
    [25] Kull M, Silva Filho TM, Flach P (2017) Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. Proc Mach Learn Res 54: 623-631. Available from: https://proceedings.mlr.press/v54/kull17a.html
    [26] Smith JW, Everhart JE, Dickson WC, et al. (1988) Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. Proc Annu Symp Comput Appl Med Care 9: 261-265. Available from: https://pmc.ncbi.nlm.nih.gov/articles/PMC2245318/
    [27] UCI Machine Learning RepositoryCDC Diabetes Health Indicators (2017). Available from: https://doi.org/10.24432/C53919
  • Reader Comments
  • © 2026 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)

Metrics

Article views(297) PDF downloads(11) Cited by(0)

Article outline

Figures and Tables

Figures(5)  /  Tables(8)

Other Articles By Authors

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog