Deploying machine learning models for credit decisions under regulatory mandates for explainability and reliability requires evaluation frameworks beyond predictive accuracy. Yet, benchmarking studies in credit risk still focus narrowly on the area under the receiver operating characteristic curve (AUC), neglecting probability calibration, uncertainty decomposition, and interpretability. We address this gap by benchmarking ten models, spanning deterministic and probabilistic families, across four evaluation dimensions on a proprietary dataset of 50, 819 customers from a Norwegian financial institution. Each customer is represented by 124 consecutive daily account balance observations and a binary 12-month default flag. Across 180 configurations combining five feature engineering modules with four resampling strategies, variational inference-based Bayesian models (Bayesian neural network, Bayesian logistic regression, convolutional-Bayesian neural network) exhibit posterior collapse on this imbalanced tabular dataset, producing near-uniform predicted probabilities and near-zero epistemic uncertainty, particularly under oversampling. Natural gradient boosting (NGB) emerges as the most effective probabilistic model, matching random forest and XGBoost in test AUC (0.80) while providing informative aleatoric uncertainty estimates and economically interpretable SHAP attributions. We further show that feature engineering and resampling effects are non-additive and model-specific, indicating both as first-order design variables in credit risk pipelines. These findings extend the literature to probabilistic evaluation dimensions and provide practitioners with guidance for model selection in regulatory-compliant credit decision systems under similar data conditions. Note that our performance figures represent exploratory upper-bound benchmarks rather than strictly unbiased out-of-sample estimates.
Citation: André Engan, Lars Ole Hjelkrem, Morten Risstad. Probabilistic versus deterministic machine learning for credit default prediction: A multidimensional benchmarking study under severe class imbalance[J]. Networks and Heterogeneous Media, 2026, 21(5): 1641-1677. doi: 10.3934/nhm.2026062
Deploying machine learning models for credit decisions under regulatory mandates for explainability and reliability requires evaluation frameworks beyond predictive accuracy. Yet, benchmarking studies in credit risk still focus narrowly on the area under the receiver operating characteristic curve (AUC), neglecting probability calibration, uncertainty decomposition, and interpretability. We address this gap by benchmarking ten models, spanning deterministic and probabilistic families, across four evaluation dimensions on a proprietary dataset of 50, 819 customers from a Norwegian financial institution. Each customer is represented by 124 consecutive daily account balance observations and a binary 12-month default flag. Across 180 configurations combining five feature engineering modules with four resampling strategies, variational inference-based Bayesian models (Bayesian neural network, Bayesian logistic regression, convolutional-Bayesian neural network) exhibit posterior collapse on this imbalanced tabular dataset, producing near-uniform predicted probabilities and near-zero epistemic uncertainty, particularly under oversampling. Natural gradient boosting (NGB) emerges as the most effective probabilistic model, matching random forest and XGBoost in test AUC (0.80) while providing informative aleatoric uncertainty estimates and economically interpretable SHAP attributions. We further show that feature engineering and resampling effects are non-additive and model-specific, indicating both as first-order design variables in credit risk pipelines. These findings extend the literature to probabilistic evaluation dimensions and provide practitioners with guidance for model selection in regulatory-compliant credit decision systems under similar data conditions. Note that our performance figures represent exploratory upper-bound benchmarks rather than strictly unbiased out-of-sample estimates.
| [1] | European Union, Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 on laying down harmonised rules on artificial intelligence and amending certain Union legislative acts (Artificial Intelligence Act), 2024. Available from: https://eur-lex.europa.eu/eli/reg/2024/1689/oj. |
| [2] | B. Supervision, Basel committee on banking supervision, Principles for Sound Liquidity Risk Management and Supervision (September 2008) (2008). Available from: https://www.bis.org/publ/bcbs144.htm. |
| [3] |
N. Q. K. Le, Y. Y. Ou, Incorporating efficient radial basis function networks and significant amino acid pairs for predicting GTP binding sites in transport proteins, BMC Bioinf., 17 (2016), 501. https://doi.org/10.1186/s12859-016-1369-y doi: 10.1186/s12859-016-1369-y
|
| [4] |
N. Q. K. Le, T. T. D. Nguyen, Y. Y. Ou, Identifying the molecular functions of electron transport proteins using radial basis function networks and biochemical properties, J. Mol. Graphics Modell., 73 (2017), 166–178. https://doi.org/10.1016/j.jmgm.2017.01.003 doi: 10.1016/j.jmgm.2017.01.003
|
| [5] |
O. N. Nymo, M. Risstad, U. T. Sjøli, T. B. Sæther, Revisiting the spanning hypothesis with machine learning methods: A systematic literature review, J. Int. Money Financ., 167 (2026), 103632. https://doi.org/10.1016/j.jimonfin.2026.103632 doi: 10.1016/j.jimonfin.2026.103632
|
| [6] | S. Eggen, T. J. Espe, K. Grude, M. Risstad, R. Sandberg, Financial time series uncertainty: A review of probabilistic AI applications, J. Econ. Surv., (2025). https://doi.org/10.1111/joes.70018 |
| [7] |
E. S. Gunnarsson, H. R. Isern, A. Kaloudis, M. Risstad, B. Vigdel, S. Westgaard, Prediction of realized volatility and implied volatility indices using AI and machine learning: A review, Int. Rev. Financ. Anal., 93 (2024), 103221. https://doi.org/10.1016/j.irfa.2024.103221 doi: 10.1016/j.irfa.2024.103221
|
| [8] |
S. Lessmann, B. Baesens, H. V. Seow, L. C. Thomas, Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research, Eur. J. Oper. Res., 247 (2015), 124–136. https://doi.org/10.1016/j.ejor.2015.05.030 doi: 10.1016/j.ejor.2015.05.030
|
| [9] |
A. Engan, L. O. Hjelkrem, M. Risstad, A review of credit risk assessment in banking-from deterministic models to probabilistic AI, Finance Res. Open, 2 (2026), 100125. https://doi.org/10.1016/j.finr.2026.100125 doi: 10.1016/j.finr.2026.100125
|
| [10] |
Z. Ghahramani, Probabilistic machine learning and artificial intelligence, Nature, 521 (2015), 452–459. https://doi.org/10.1038/nature14541 doi: 10.1038/nature14541
|
| [11] | Y. Gal, Z. Ghahramani, Dropout as a Bayesian approximation: Representing model uncertainty in deep learning, in International Conference on Machine Learning, PMLR, (2016), 1050–1059. Available from: https://proceedings.mlr.press/v48/gal16.html. |
| [12] |
Y. Wang, Y. Zhang, Y. Lu, X. Yu, A comparative assessment of credit risk model based on machine learning–-A case study of bank loan data, Procedia Comput. Sci., 174 (2020), 141–149. https://doi.org/10.1016/j.procs.2020.06.069 doi: 10.1016/j.procs.2020.06.069
|
| [13] | Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, et al., Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift, Adv. Neural Inf. Process. Syst., 32 (2019). Available from: https://proceedings.neurips.cc/paper_files/paper/2019/file/8558cb408c1d76621371888657d2eb1d-Paper.pdf. |
| [14] |
M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, et al., A review of uncertainty quantification in deep learning: Techniques, applications and challenges, Inf. Fusion, 76 (2021), 243–297. https://doi.org/10.1016/j.inffus.2021.05.008 doi: 10.1016/j.inffus.2021.05.008
|
| [15] | D. J. Hand, W. E. Henley, Statistical classification methods in consumer credit scoring: A review, J. R. Stat. Soc. Ser. A, 160 (1997), 523–541. Available from: https://www.jstor.org/stable/2983268 |
| [16] |
J. N. Crook, D. B. Edelman, L. C. Thomas, Recent developments in consumer credit risk assessment, Eur. J. Oper. Res., 183 (2007), 1447–1465. https://doi.org/10.1016/j.ejor.2006.09.100 doi: 10.1016/j.ejor.2006.09.100
|
| [17] | I. R. Berrada, F. Barramou, O. B. Alami, Towards a machine learning-based model for corporate loan default prediction, Int. J. Adv. Comput. Sci. Appl., 15 (2024). https://doi.org/10.14569/IJACSA.2024.0150357 |
| [18] |
H. Kvamme, N. Sellereite, K. Aas, S. Sjursen, Predicting mortgage default using convolutional neural networks, Expert Syst. Appl., 102 (2018), 207–217. https://doi.org/10.1016/j.eswa.2018.02.029 doi: 10.1016/j.eswa.2018.02.029
|
| [19] |
L. O. Hjelkrem, P. E. De Lange, E. Nesset, An end-to-end deep learning approach to credit scoring using {CNN+ XGBoost} on transaction data, J. Risk Model Valid., 16 (2022), 1–25. https://doi.org/10.21314/JRMV.2022.018 doi: 10.21314/JRMV.2022.018
|
| [20] |
M. C. Aniceto, F. Barboza, H. Kimura, Machine learning predictivity applied to consumer creditworthiness, Future Bus. J., 6 (2020), 37. https://doi.org/10.1186/s43093-020-00041-w doi: 10.1186/s43093-020-00041-w
|
| [21] |
L. Coenen, W. Verbeke, T. Guns, Machine learning methods for short-term probability of default: A comparison of classification, regression and ranking methods, J. Oper. Res. Soc., 73 (2022), 191–206. https://doi.org/10.1080/01605682.2020.1865847 doi: 10.1080/01605682.2020.1865847
|
| [22] |
P. E. De Lange, B. Melsom, C. B. Vennerød, S. Westgaard, Explainable AI for credit assessment in banks, J. Risk Financ. Manag., 15 (2022), 556. https://doi.org/10.3390/jrfm15120556 doi: 10.3390/jrfm15120556
|
| [23] |
X. Zhu, Q. Chu, X. Song, P. Hu, L. Peng, Explainable prediction of loan default based on machine learning models, Data Sci. Manag., 6 (2023), 123–133. https://doi.org/10.1016/j.dsm.2023.04.003 doi: 10.1016/j.dsm.2023.04.003
|
| [24] | N. G. B. Abrahamsen, E. Nylén-Forthun, M. Møller, P. E. de Lange, M. Risstad, Financial distress prediction in the nordics: Early warnings from machine learning models, J. Risk Financ. Manag., 17 (2024), 432. https://www.mdpi.com/1911-8074/17/10/432 |
| [25] | A. Kendall, Y. Gal, What uncertainties do we need in Bayesian deep learning for computer vision, Adv. Neural Inf. Process. Syst., 30 (2017), 5580–5590. Available from: https://proceedings.neurips.cc/paper_files/paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf. |
| [26] | C. Blundell, J. Cornebise, K. Kavukcuoglu, D. Wierstra, Weight uncertainty in neural network, in International Conference on Machine Learning, PMLR, (2015), 1613–1622. Available from: http://proceedings.mlr.press/v37/blundell15.pdf. |
| [27] | R. Mbuvha, I. Boulkaibet, T. Marwala, Automatic relevance determination Bayesian neural networks for credit card default modelling, preprint, arXiv: 1906.06382, 2019. https://doi.org/10.48550/arXiv.1906.06382 |
| [28] | C. Louizos, M. Welling, Multiplicative normalizing flows for variational Bayesian neural networks, in International Conference on Machine Learning, PMLR, (2017), 2218–2227. Available from: http://proceedings.mlr.press/v70/louizos17a/louizos17a.pdf. |
| [29] | A. Ashukha, A. Lyzhov, D. Molchanov, D. Vetrov, Pitfalls of in-domain uncertainty estimation and ensembling in deep learning, preprint, arXiv: 2002.06470, 2021. https://doi.org/10.48550/arXiv.2002.06470 |
| [30] |
J. Černevičienė, A. Kabašinskas, Explainable artificial intelligence (XAI) in finance: A systematic literature review, Artif. Intell. Rev., 57 (2024), 216. https://doi.org/10.1007/s10462-024-10854-8 doi: 10.1007/s10462-024-10854-8
|
| [31] |
C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat. Mach. Intell., 1 (2019), 206–215. https://doi.org/10.1038/s42256-019-0048-x doi: 10.1038/s42256-019-0048-x
|
| [32] |
M. Nagl, M. Nagl, D. Rösch, Quantifying uncertainty of machine learning methods for loss given default, Front. Appl. Math. Stat., 8 (2022), 1076083. https://doi.org/10.3389/fams.2022.1076083 doi: 10.3389/fams.2022.1076083
|
| [33] | A. Niculescu-Mizil, R. Caruana, Predicting good probabilities with supervised learning, in Proceedings of the 22nd International Conference on Machine Learning, PMLR, (2005), 625–632. https://doi.org/10.1145/1102351.1102430 |
| [34] | C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On calibration of modern neural networks, in International Conference on Machine Learning, PMLR, (2017), 1321–1330. Available from: http://proceedings.mlr.press/v70/guo17a/guo17a.pdf. |
| [35] | S. Lundberg, S. Lee, A unified approach to interpreting model predictions, in Advances in Neural Information Processing Systems 30, (2017), NIPS. |
| [36] |
Y. Chen, R. Calabrese, B. Martin-Barragan, Interpretable machine learning for imbalanced credit scoring datasets, Eur. J. Oper. Res., 312 (2024), 357–372. https://doi.org/10.1016/j.ejor.2023.06.036 doi: 10.1016/j.ejor.2023.06.036
|
| [37] | T. Hastie, R. Tibshirani, J. H. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, 2009. Available from: https://link.springer.com/book/10.1007/978-0-387-84858-7. |
| [38] |
N. V. Chawla, K. W. Bowyer, L. O. Hall, W. P. Kegelmeyer, SMOTE: Synthetic minority over-sampling technique, J. Artif. Intell. Res., 16 (2002), 321–357. https://doi.org/10.48550/arXiv.1106.1813 doi: 10.48550/arXiv.1106.1813
|
| [39] |
H. He, E. A. Garcia, Learning from imbalanced data, IEEE Trans. Knowl. Data Eng., 21 (2009), 1263–1284. https://doi.org/10.1109/TKDE.2008.239 doi: 10.1109/TKDE.2008.239
|
| [40] | K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Available from: https://mitpress.mit.edu/9780262046824/probabilistic-machine-learning/. |
| [41] | L. Breiman, Random forests, Mach. Learn., 45 (2001), 5–32. https://doi.org/10.1023/A: 1010933404324 |
| [42] | T. Chen, C. Guestrin, XGBoost: A scalable tree boosting system, in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, (2016), 785–794. https://doi.org/10.1145/2939672.2939785 |
| [43] | I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. Available from: https://mitpress.mit.edu/9780262035613/deep-learning/. |
| [44] | K. P. Murphy, Probabilistic Machine Learning: Advanced Topics, MIT Press, 2023. Available from: https://mitpress.mit.edu/9780262048439/probabilistic-machine-learning/. |
| [45] | T. Duan, A. Anand, D. Y. Ding, K. K. Thai, S. Basu, A. Ng, et al., NGBoost: Natural gradient boosting for probabilistic prediction, in International Conference on Machine Learning, PMLR, (2020), 2690–2700. Available from: http://proceedings.mlr.press/v119/duan20a/duan20a.pdf. |
| [46] | G. W. Brier, Verification of forecasts expressed in terms of probability, Mon. Weather Rev., 78 (1950), 1–3. Available from: https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml. |
| [47] |
T. Saito, M. Rehmsmeier, The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, PLoS One, 10 (2015), 1–21. https://doi.org/10.1371/journal.pone.0118432 doi: 10.1371/journal.pone.0118432
|
| [48] | B. Baesens, R. Setiono, C. Mues, J. Vanthienen, Using neural network rule extraction and decision tables for credit-risk evaluation, Manage. Sci., 49 (2003), 312–329. Available from: https://www.jstor.org/stable/4133928. |
| [49] |
R. Shwartz-Ziv, A. Armon, Tabular data: Deep learning is not all you need, Inf. Fusion, 81 (2022), 84–90. https://doi.org/10.1016/j.inffus.2021.11.011 doi: 10.1016/j.inffus.2021.11.011
|
| [50] | J. Van Amersfoort, L. Smith, Y. W. Teh, Y. Gal, Uncertainty estimation using a single deep deterministic neural network, preprint, arXiv: 2003.02037, 2020. https://doi.org/10.48550/arXiv.2003.02037 |
| [51] | A. Alemi, B. Poole, I. Fischer, J. Dillon, R. A. Saurous, K. Murphy, Fixing a broken ELBO, preprint, arXiv: 1711.00464, 2018. https://doi.org/10.48550/arXiv.1711.00464 |
| [52] | G. Incorvaia, D. Hond, H. Asgari, Uncertainty quantification for machine learning output assurance using anomaly-based dataset dissimilarity measures, in 2023 IEEE International Conference On Artificial Intelligence Testing (AITest), IEEE, (2023), 26–33. https://doi.org/10.1109/AITest58265.2023.00013 |
| [53] |
E. I. Altman, Financial ratios, discriminant analysis and the prediction of corporate bankruptcy, J. Finance, 23 (1968), 589–609. https://doi.org/10.2307/2978933 doi: 10.2307/2978933
|
| [54] | L. Thomas, J. Crook, D. Edelman, Credit Scoring and Its Applications, SIAM, 2017. Available from: https://epubs.siam.org/doi/book/10.1137/1.9781611974560. |
| [55] | M. Dusenberry, G. Jerfel, Y. Wen, Y. Ma, J. Snoek, K. Heller, et al., Efficient and scalable Bayesian neural nets with rank-1 factors, preprint, arXiv: 2005.07186, 2020. https://doi.org/10.48550/arXiv.2005.07186 |
| [56] | P. Izmailov, S. Vikram, M. D. Hoffman, A. G. G. Wilson, What are Bayesian neural network posteriors really like, in International Conference on Machine Learning, PMLR, (2021), 4629–4640. Available from: http://proceedings.mlr.press/v139/izmailov21a/izmailov21a.pdf. |
| [57] | A. G. Wilson, P. Izmailov, Bayesian deep learning and a probabilistic perspective of generalization, preprint, arXiv: 2002.08791, 2022. https://doi.org/10.48550/arXiv.2002.08791 |
| [58] | B. Dai, D. Wipf, Diagnosing and enhancing VAE models, preprint, arXiv: 1903.05789, 2019. https://doi.org/10.48550/arXiv.1903.05789 |
| [59] | J. He, D. Spokoyny, G. Neubig, T. Berg-Kirkpatrick, Lagging inference networks and posterior collapse in variational autoencoders, preprint, arXiv: 1901.05534, 2019. https://doi.org/10.48550/arXiv.1901.05534 |
| [60] |
E. R. DeLong, D. M. DeLong, D. L. Clarke-Pearson, Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach, Biometrics, 44 (1988), 837–845. https://doi.org/10.2307/2531595 doi: 10.2307/2531595
|
| [61] |
R. J. Tibshirani, B. Efron, An introduction to the bootstrap, Monogr. Stat. Appl. Probab., 57 (1993), 1–436. https://doi.org/10.1201/9780429246593 doi: 10.1201/9780429246593
|
Figures(9) / Tables(10)
André Engan, Lars Ole Hjelkrem, Morten Risstad. Probabilistic versus deterministic machine learning for credit default prediction: A multidimensional benchmarking study under severe class imbalance[J]. Networks and Heterogeneous Media, 2026, 21(5): 1641-1677. doi: 10.3934/nhm.2026062
DownLoad: