In this study we investigate alternative interpretable machine learning ("IML") models in the context of probability of default ("PD") modeling for the large corporate asset class. IML models have become increasingly prominent in highly regulated industries where there are concerns over the unintended consequences of deploying black box models that may be deemed conceptually unsound. In the context of banking and in wholesale portfolios, there are challenges around using models where the outcomes may not be explainable, both in terms of the business use case as well as meeting model validation standards. We compare various IML models (deep neural networks and explainable boosting machines), including standard approaches such as logistic regression, using a long and robust history of corporate borrowers. We find that there are material differences between the approaches in terms of dimensions such as model predictive performance and the importance or robustness of risk factors in driving outcomes, including conflicting conclusions depending upon the IML model and the benchmarking measure considered. These findings call into question the value of the modest pickup in performance with the IML models relative to a more traditional technique, especially if these models are to be applied in contexts that must meet supervisory and model validation standards.
Citation: Michael Jacobs, Jr. Benchmarking alternative interpretable machine learning models for corporate probability of default[J]. Data Science in Finance and Economics, 2024, 4(1): 1-52. doi: 10.3934/DSFE.2024001
| [1] | Yuan Tian, Sanyi Tang . Dynamics of a density-dependent predator-prey biological system with nonlinear impulsive control. Mathematical Biosciences and Engineering, 2021, 18(6): 7318-7343. doi: 10.3934/mbe.2021362 |
| [2] | Gianni Gilioli, Sara Pasquali, Fabrizio Ruggeri . Nonlinear functional response parameter estimation in a stochastic predator-prey model. Mathematical Biosciences and Engineering, 2012, 9(1): 75-96. doi: 10.3934/mbe.2012.9.75 |
| [3] | Qizhen Xiao, Binxiang Dai . Heteroclinic bifurcation for a general predator-prey model with Allee effect and state feedback impulsive control strategy. Mathematical Biosciences and Engineering, 2015, 12(5): 1065-1081. doi: 10.3934/mbe.2015.12.1065 |
| [4] | Jie Hu, Juan Liu, Peter Yuen, Fuzhong Li, Linqiang Deng . Modelling of a seasonally perturbed competitive three species impulsive system. Mathematical Biosciences and Engineering, 2022, 19(3): 3223-3241. doi: 10.3934/mbe.2022149 |
| [5] | Zhenzhen Shi, Huidong Cheng, Yu Liu, Yanhui Wang . Optimization of an integrated feedback control for a pest management predator-prey model. Mathematical Biosciences and Engineering, 2019, 16(6): 7963-7981. doi: 10.3934/mbe.2019401 |
| [6] | Eric M. Takyi, Kasey Cooper, Ava Dreher, Caroline McCrorey . The (De)Stabilizing effect of juvenile prey cannibalism in a stage-structured model. Mathematical Biosciences and Engineering, 2023, 20(2): 3355-3378. doi: 10.3934/mbe.2023158 |
| [7] | Saheb Pal, Nikhil Pal, Sudip Samanta, Joydev Chattopadhyay . Fear effect in prey and hunting cooperation among predators in a Leslie-Gower model. Mathematical Biosciences and Engineering, 2019, 16(5): 5146-5179. doi: 10.3934/mbe.2019258 |
| [8] | Eric M. Takyi, Charles Ohanian, Margaret Cathcart, Nihal Kumar . Dynamical analysis of a predator-prey system with prey vigilance and hunting cooperation in predators. Mathematical Biosciences and Engineering, 2024, 21(2): 2768-2786. doi: 10.3934/mbe.2024123 |
| [9] | Jinxing Zhao, Yuanfu Shao . Bifurcations of a prey-predator system with fear, refuge and additional food. Mathematical Biosciences and Engineering, 2023, 20(2): 3700-3720. doi: 10.3934/mbe.2023173 |
| [10] | Shunyi Li . Hopf bifurcation, stability switches and chaos in a prey-predator system with three stage structure and two time delays. Mathematical Biosciences and Engineering, 2019, 16(6): 6934-6961. doi: 10.3934/mbe.2019348 |
In this study we investigate alternative interpretable machine learning ("IML") models in the context of probability of default ("PD") modeling for the large corporate asset class. IML models have become increasingly prominent in highly regulated industries where there are concerns over the unintended consequences of deploying black box models that may be deemed conceptually unsound. In the context of banking and in wholesale portfolios, there are challenges around using models where the outcomes may not be explainable, both in terms of the business use case as well as meeting model validation standards. We compare various IML models (deep neural networks and explainable boosting machines), including standard approaches such as logistic regression, using a long and robust history of corporate borrowers. We find that there are material differences between the approaches in terms of dimensions such as model predictive performance and the importance or robustness of risk factors in driving outcomes, including conflicting conclusions depending upon the IML model and the benchmarking measure considered. These findings call into question the value of the modest pickup in performance with the IML models relative to a more traditional technique, especially if these models are to be applied in contexts that must meet supervisory and model validation standards.
| [1] |
Abdulrahman UFI, Panford JK, Hayfron-Acquah JB (2014) Fuzzy logic approach to credit scoring for micro finances in Ghana: a case study of KWIQPUS money lending. Int J Comput Appl 94: 11–18. https://doi.org/10.5120/16362-5772 doi: 10.5120/16362-5772
|
| [2] | Vahid PR, Ahmadi A (2016) Modeling corporate customers' credit risk considering the ensemble approaches in multiclass classification: evidence from Iranian corporate credits. J Credit Risk 12: 71–95. |
| [3] | Allen L, Peng L, Shan Y (2020) Social networks and credit allocation on fintech lending platforms. working paper, social science research network. Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id = 3537714. |
| [4] | Altman EI (1968) Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. J Financ 23: 589–609. Available from: https://pdfs.semanticscholar.org/cab5/059bfc5bf4b70b106434e0cb665f3183fd4a.pdf. |
| [5] | Altman EI, Narayanan P (1997) An international survey of business failure classification models. in financial markets, institutions and instruments. New York: New York University Salomon Center, 6. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/1468-0416.00010. |
| [6] | American Banking Association (2018) New credit score unveiled drawing on bank account data. ABA Bank J, October 22. |
| [7] |
Anagnostou I, Kandhai D, Sanchez Rivero J, et al. (2020) Contagious defaults in a credit portfolio: a Bayesian network approach. J Credit Risk 16: 1–26. https://dx.doi.org/10.2139/ssrn.3446615 doi: 10.2139/ssrn.3446615
|
| [8] |
Bjorkegren D, Grissen D (2020) Behavior revealed in mobile phone usage predicts credit repayment. World Bank Econ Rev 34: 618–634. https://doi.org/10.1093/wber/lhz006 doi: 10.1093/wber/lhz006
|
| [9] | Bonds D (1999) Modeling term structures of defaultable bonds. Rev Financ Stud 12: 687–720. Available from: https://academic.oup.com/rfs/article-abstract/12/4/687/1578719?redirectedFrom = fulltext. |
| [10] | Breeden J (2021) A survey of machine learning in credit risk. J Risk Model Validat 17: 1–62. |
| [11] | Chava S, Jarrow RA (2004) Bankruptcy prediction with industry effects. Rev Financ 8: 537–69. Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id = 287474. |
| [12] | Clemen R (1989) Combining forecasts: a review and annotated bibliography. Int J Forecast 5: 559–583 Available from: https://people.duke.edu/~clemen/bio/Published%20Papers/13.CombiningReview-Clemen-IJOF-89.pdf. |
| [13] | Coats PK, Fant LF (1993) Recognizing financial distress patterns using a neural network tool. Financ Manage 22: 142–155. Available from: https://ideas.repec.org/a/fma/fmanag/coats93.html. |
| [14] | Collobert R, Weston J (2008) A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Neural Information Processing Systems, 1: 160–167. Association for Computing Machinery, New York. https://doi.org/10.1145/1390156.1390177 |
| [15] | Duffie D, Singleton KJ (1999) Simulating correlated defaults. Paper presented at the Bank of England Conference on Credit Risk Modeling and Regulatory Implications Working Paper, Stanford University. Available from: https://kenneths.people.stanford.edu/sites/g/files/sbiybj3396/f/duffiesingleton1999.pdf. |
| [16] | Dwyer DW, Kogacil AE, Stein RM (2004) Moody's KMV RiskCalcTM v2.1 Model. Moody's Analytics. Available from: https://www.moodys.com/sites/products/productattachments/riskcalc%202.1%20whitepaper.pdf. |
| [17] | Harrell FE Jr (2018) Road Map for Choosing Between Statistical Modeling and Machine Learning, Stat Think. Available from: http://www.fharrell.com/post/stat-ml. |
| [18] |
Jacobs Jr M (2022a) Quantification of model risk with an application to probability of default estimation and stress testing for a large corporate portfolio. J Risk Model Validat 15: 1–39. https://doi.org/10.21314/JRMV.2022.023 doi: 10.21314/JRMV.2022.023
|
| [19] |
Jacobs Jr M (2022b) Validation of corporate probability of default models considering alternative use cases and the quantification of model risk. Data Sci Financ Econ 2: 17–53. https://doi.org/10.3934/DSFE.2022002 doi: 10.3934/DSFE.2022002
|
| [20] |
Jarrow RA, Turnbull SM (1995) Pricing derivatives on financial securities subject to credit risk. J Fnanc 50: 53–85. https://doi.org/10.1111/j.1540-6261.1995.tb05167.x doi: 10.1111/j.1540-6261.1995.tb05167.x
|
| [21] | Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, 1: 1097–1105. Available from: https://cse.iitk.ac.in/users/cs365/2013/hw2/krizhevsky-hinton-12-imagenet-convolutional-NN-deep.pdf. |
| [22] | Kumar IE, Venkatasubramanian S, Scheidegger C, et al. (2020) Problems with Shapley-Value-Based Explanations as Feature Importance Measures. In International Conference on Machine Learning Research: 5491–5500. Available from: https://proceedings.mlr.press/v119/kumar20e.html. |
| [23] |
Lessmann S, Baesens B, Seow HV, et al. (2015) Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research. Europ J Oper Res 247: 124–136. https://doi.org/10.1016/j.ejor.2015.05.030 doi: 10.1016/j.ejor.2015.05.030
|
| [24] |
Li K, Niskanen J, Kolehmainen M, et al. (2016) Financial innovation: Credit default hybrid model for SME lending. Expert Syst Appl 61: 343–355. https://doi.org/10.1016/j.eswa.2016.05.029 doi: 10.1016/j.eswa.2016.05.029
|
| [25] | Li X, Liu S, Li Z, et al. (2020) Flowscope: spotting money laundering based on graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, 34: 4731–4738. https://doi.org/10.1609/aaai.v34i04.5906 |
| [26] | Lou Y, Caruana R, Gehrke J, et al. (2013) Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 623–631. https://dl.acm.org/doi/abs/10.1145/2487575.2487579 |
| [27] | Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 4768–4777. Available from: https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html. |
| [28] | Mckee TE (2000) Developing a bankruptcy prediction model via rough sets theory. Intell Syst Account Financ Manage 9: 159–173. https://doi.org/fv22ks |
| [29] | Merton RC (1974) On the pricing of corporate debt: The risk structure of interest rates. J Fnanc 29: 449–470. |
| [30] | Mester LJ (1997) What's the point of credit scoring? Federal Reserve Bank of Philadelphia. Bus Rev 3: 3–16. Available from: https://fraser.stlouisfed.org/files/docs/historical/frbphi/businessreview/frbphil_rev_199709.pdf. |
| [31] |
Min JH, Lee YC (2005) Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters. Expert Syst Appl 28: 603–614 https://doi.org/10.1016/j.eswa.2004.12.008 doi: 10.1016/j.eswa.2004.12.008
|
| [32] | Molnar C, König G, Herbinger J, et al. (2020) Pitfalls to avoid when interpreting machine learning models. Working Paper, University of Vienna. Available from: http://eprints.cs.univie.ac.at/6427/. |
| [33] | Odom MD, Sharda R (1990) A neural network model for bankruptcy prediction. In Joint Conference on Neural Networks, 163–168. IEEE Press, Piscataway, NJ. Available from: https://ieeexplore.ieee.org/abstract/document/5726669. |
| [34] | Opitz D, Maclin R (1999) Popular ensemble methods: An empirical study. J Artif Intell Res 11: 169–198. |
| [35] | Ribeiro MT, Singh S, Guestrin C (2016) "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 1135–1144. Available from: https://dl.acm.org/doi/abs/10.1145/2939672.2939778. |
| [36] |
Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1: 206–215. https://doi.org/10.1038/s42256-019-0048-x doi: 10.1038/s42256-019-0048-x
|
| [37] | Slack D, Hilgard S, Jia E, et al. (2020) Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 180–186. https://dl.acm.org/doi/abs/10.1145/3375627.3375830 |
| [38] | Sudjianto A, Knauth W, Rahul S, et al. (2020) Unwrapping the black box of deep ReLU networks: Interpretability, diagnostics, and simplification. arXiv preprin, Cornell University. https://arXiv.org/abs/2011.04041 |
| [39] | Sudjianto A, Zhang A (2021) Designing inherently interpretable machine learning models. In Proceedings of ACM ICAIF 2021 Workshop on Explainable AI in Finance. ACM, New York. https://arXiv.org/abs/2111.01743 |
| [40] | Sudjianto A, Zhang A, Yang Z, et al. (2023) PiML toolbox for interpretable machine learning model development and validation. arXiv preprint arXiv. https://doi.org/10.48550/arXiv.2305.04214 |
| [41] | U.S. Banking Regulatory Agencies (2011) The U.S. Office of the Comptroller of the Currency and the Board of Governors of Federal Reserve System. SR 11-7/OCC11-12: Supervisory Guidance on Model Risk Management. Washington, D.C. Available from: https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf. |
| [42] | U.S. Banking Regulatory Agencies (2021) The U.S. Office of Comptroller of the Currency, the Board of Governors of the Federal Reserve System, the Federal Deposit Insurance Corporation, the Consumer Financial Protection Bureau, and the National Credit Union Administration. Request for Information and Comment on Financial Institutions' Use of Artificial Intelligence, Including Machine Learning. Washington, D.C. Available from: https://www.federalregister.gov/documents/2021/03/31/2021-06607/requestfor-information-and-comment-on-financial-institutions-use-of-artificialintelligence. |
| [43] | U.S. Office of the Comptroller of the Currency (2021) Comptroller's Handbook on Model Risk Management. Washington, D.C. Available from: https://www.occ.treas.gov/publicationsand-resources/publications/comptrollers-handbook/files/model-riskmanagement/index-model-risk-management.html. |
| [44] |
Vassiliou PC (2013) Fuzzy semi-Markov migration process in credit risk. Fuzzy Sets and Syst 223: 39–58. https://doi.org/10.1016/j.fss.2013.02.016 doi: 10.1016/j.fss.2013.02.016
|
| [45] |
Yang Z, Zhang A, Sudjianto A (2021a) Enhancing explainability of neural networks through architecture constraints. IEEE T Neur Net Learn Syst 32: 2610–2621. https://doi.org/10.1109/TNNLS.2020.3007259 doi: 10.1109/TNNLS.2020.3007259
|
| [46] |
Yang Z, Zhang A, Sudjianto A (2021) GAMI-Net: An explainable neural network based on generalized additive models with structured interactions. Pattern Recogn 120: 108192. https://doi.org/10.1016/j.patcog.2021.10819 doi: 10.1016/j.patcog.2021.10819
|
| [47] |
Zhu Y, Xie C, Wang G J, et al. (2017) Comparison of individual, ensemble and integrated ensemble machine learning methods to predict China's SME credit risk in supply chain finance. Neural Comput Appl 28: 41–50. https://doi.org/10.1007/s00521-016-2304-x doi: 10.1007/s00521-016-2304-x
|
| 1. | Weijian Xu, Lansun Chen, Shidong Chen, Guoping Pang, An impulsive state feedback control model for releasing white-headed langurs in captive to the wild, 2016, 34, 10075704, 199, 10.1016/j.cnsns.2015.10.015 | |
| 2. | Huidong Cheng, Fang Wang, Tongqian Zhang, Multi-State Dependent Impulsive Control for Pest Management, 2012, 2012, 1110-757X, 1, 10.1155/2012/381503 | |
| 3. | HONG ZHANG, RONGPING ZHU, LANSUN CHEN, ON A PERIODIC TIME-DEPENDENT IMPULSIVE SYSTEM OF STRATEGIES FOR CONTROLLING THE APPLE SNAIL IN PADDY FIELDS, 2007, 15, 0218-3390, 397, 10.1142/S0218339007002271 | |
| 4. | Hunki Baek, Dynamic Complexities of a Three-Species Beddington-DeAngelis System with Impulsive Control Strategy, 2010, 110, 0167-8019, 23, 10.1007/s10440-008-9378-0 | |
| 5. | BING LIU, YE TIAN, BAOLIN KANG, DYNAMICS ON A HOLLING II PREDATOR–PREY MODEL WITH STATE-DEPENDENT IMPULSIVE CONTROL, 2012, 05, 1793-5245, 1260006, 10.1142/S1793524512600066 | |
| 6. | Kaibiao Sun, Shan Liu, Andrzej Kasperski, Yuan Tian, Dynamics Analysis and Biomass Productivity Optimisation of a Microbial Cultivation Process through Substrate Regulation, 2016, 2016, 1026-0226, 1, 10.1155/2016/3685941 | |
| 7. | Xiangsen Liu, Binxiang Dai, The dynamics of a stage structure population model with fixed-time birth pulse and state feedback control strategy, 2016, 2016, 1687-1847, 10.1186/s13662-016-0852-0 | |
| 8. | Huidong Cheng, Fang Wang, Tongqian Zhang, Multi-State Dependent Impulsive Control for Holling I Predator-Prey Model, 2012, 2012, 1026-0226, 1, 10.1155/2012/181752 | |
| 9. | Shidong Chen, Weijian Xu, Lansun Chen, Zhonghao Huang, A White-headed langurs impulsive state feedback control model with sparse effect and continuous delay, 2017, 50, 10075704, 88, 10.1016/j.cnsns.2017.02.003 | |
| 10. | XINZHU MENG, ZHITAO SONG, LANSUN CHEN, A NEW MATHEMATICAL MODEL FOR OPTIMAL CONTROL STRATEGIES OF INTEGRATED PEST MANAGEMENT, 2007, 15, 0218-3390, 219, 10.1142/S0218339007002143 | |
| 11. | Tongqian Zhang, Wanbiao Ma, Xinzhu Meng, Impulsive control of a continuous-culture and flocculation harvest chemostat model, 2017, 48, 0020-7721, 3459, 10.1080/00207721.2017.1384861 | |
| 12. | Hong Zhang, Lansun Chen, Juan J. Nieto, A delayed epidemic model with stage-structure and pulses for pest management strategy, 2008, 9, 14681218, 1714, 10.1016/j.nonrwa.2007.05.004 | |
| 13. | LICHUN ZHAO, JINGNA LIU, WEI GAO, BIFURCATION CONTROL OF A PREDATOR–PREY MODEL BASED ON NUTRITION KINETICS, 2013, 06, 1793-5245, 1350019, 10.1142/S1793524513500198 | |
| 14. | Xuehui Ji, Sanling Yuan, Lansun Chen, A pest control model with state-dependent impulses, 2015, 08, 1793-5245, 1550009, 10.1142/S1793524515500096 | |
| 15. | YANKE DU, RUI XU, LIJIANG DUAN, DYNAMICS OF A STAGE-STRUCTURED PREDATOR-PREY MODEL CONCERNING IMPULSIVE CONTROL STRATEGY, 2009, 17, 0218-3390, 779, 10.1142/S0218339009003046 | |
| 16. | Zhongjun Ma, Jian Yang, Guirong Jiang, Impulsive control in a stage structure population model with birth pulses, 2010, 217, 00963003, 3453, 10.1016/j.amc.2010.09.012 | |
| 17. | Hunki Baek, The Dynamics of a Predator-Prey System with State-Dependent Feedback Control, 2012, 2012, 1085-3375, 1, 10.1155/2012/101386 | |
| 18. | Huidong Cheng, Tongqian Zhang, Fang Wang, Existence and Attractiveness of Order One Periodic Solution of a Holling I Predator-Prey Model, 2012, 2012, 1085-3375, 1, 10.1155/2012/126018 | |
| 19. | Baolin Kang, Mingfeng He, Bing Liu, Optimal Control of Agricultural Insects with a Stage-Structured Model, 2013, 2013, 1024-123X, 1, 10.1155/2013/168979 | |
| 20. | THE GEOMETRICAL ANALYSIS OF A PREDATOR-PREY MODEL WITH MULTI-STATE DEPENDENT IMPULSES, 2018, 8, 2156-907X, 427, 10.11948/2018.427 | |
| 21. | Hunki Baek, Extinction and Permanence of a Three-Species Lotka-Volterra System with Impulsive Control Strategies, 2008, 2008, 1026-0226, 1, 10.1155/2008/752403 | |
| 22. | Linfei Nie, Zhidong Teng, Lin Hu, Jigen Peng, The dynamics of a Lotka–Volterra predator–prey model with state dependent impulsive harvest for predator, 2009, 98, 03032647, 67, 10.1016/j.biosystems.2009.06.001 | |
| 23. | A. K. Misra, Akash Yadav, Dynamics of a stage-structured insect–vegetable crop interaction model with maturation delay, 2025, 18, 1793-5245, 10.1142/S1793524524500037 | |
| 24. | Mengge Zhao, Jinyan Wang, Study on a switching model with stage structure for the integrated management of Bactericera gobica Loginova, 2025, 2025, 2731-4235, 10.1186/s13662-025-03939-3 |
Figures(67) / Tables(7)
Michael Jacobs, Jr. Benchmarking alternative interpretable machine learning models for corporate probability of default[J]. Data Science in Finance and Economics, 2024, 4(1): 1-52. doi: 10.3934/DSFE.2024001
DownLoad: