Prediction of bank credit customers churn based on machine learning and interpretability analysis

1.   Introduction

In the financial market, risk management has become one of the key elements of banking. Credit card customer churn prediction plays an important role in the risk management of the banking industry, assisting banks in identifying potential risks and implementing suitable measures to mitigate them (Butaru et al., 2016). With the development of technology, the influence of customer churn prediction has extended into various fields, including the financial industry. In particular, the rapid progression of machine learning theories and algorithms has offered a fresh perspective and solution for customer churn prediction (Alfaiz and Fati, 2022). Firstly, machine learning technology is capable of automatically extracting useful information from large amounts of historical data, identifying distribution patterns and relationships to predict future trends and behaviors (Pudjihartono et al., 2022). In credit card customer churn prediction, machine learning methods can efficiently identify key variables and underlying relationships related to customer attrition. By analyzing the mass multi-dimensional data, including consumer behavior, credit history, and demographic information of historical users, machine learning can construct an effective prediction model. Secondly, machine learning technology has strong generalization ability, automatically adapting to new and unseen data after training without frequent adjustments and optimizations (Janiesch et al., 2021). Bank customer churn prediction using machine learning methods presents both challenges and opportunities. Through comprehensive studies and the application of machine learning methods, prediction models provide more precise and reliable results, thereby enhancing the accuracy and reliability of risk management standards in the credit card business and supporting the sustainable development of banking.

Despite the significant advantages of machine learning in customer churn prediction, its practical implementation still encounters several challenges. As datasets grow larger and models become more complex, ensuring model interpretability becomes a crucial concern (Chen and Meng, 2020). The SHAP values method is applied to interpret the outcomes of machine learning models. It quantifies the importance of each variable in the dataset, considering both the independent influence of each variable on prediction outcomes and the interactions among different variables. The SHAP values method provides a comprehensive explanation of machine learning model predictions (Gebreyesus et al., 2023). Moreover, the causal inference method has gained significant popularity in recent years for data analysis and machine learning-related research. By analyzing causal effects among variables, the causal inference method excludes variables that are related to prediction results but lack actual causal connections, and filters out variables with a causal impact on prediction results. This method provides a more reliable basis for variable selection in predictions based on machine learning (Li et al., 2023). The purpose of this study is to discuss the construction of credit card customer churn prediction models by applying machine learning techniques. Additionally, the SHAP values method is utilized to complete variable selection based on the importance of influencing the outcomes and to interpret machine learning-based prediction results. The causal inference method aims to explain the prediction results by analyzing the causal relationships between variables. These causal relationships are compared with the results obtained by the SHAP values method.

The remainder of this study is organized as follows. Section 2 reviews research related to machine learning, interpretability analysis, and causal inference. Section 3 describes the concepts of four sampling techniques for data preprocessing, six machine learning methods for credit card customer churn prediction, and the causal inference method in detail. Section 4 summarizes the performance of the machine learning models, interprets the optimal model, and identifies the causal relationships by analyzing the experimental results. Finally, Section 5 presents the conclusion and describes limitations and further research.

2.   Literature review

With the development of computer hardware in recent years, technologies such as machine learning and deep learning are now widely used across various industries. Machine learning can be combined with stock price prediction to construct new quantitative investment strategies in financial investment (Yan et al., 2023). In the field of stock market risk, traditional machine learning models and neural networks can also predict Barrier Option prices with relative accuracy (Li and Yan, 2023). Even Bitcoin, a cryptocurrency that has been widely discussed in recent years, can have its price predicted by machine learning methods, with the Support Vector Regression model outperforming other models (Erfanian et al., 2022). In the area of credit card customer churn prediction, there is a growing number of reliable research papers aiming to predict churn rates by analyzing the personal and behavioral data of credit card customers, enabling banks to take effective measures to retain customers in advance. For instance, Zhang et al. used simple preprocessing of customer data and applied only the Random Forest (RF) model, a traditional machine learning method, to classify customers, achieving better classification results (Zhang et al., 2024). In the study by de Lima Lemos et al., several traditional machine learning models were used for prediction, including RF, Decision Tree, k-Nearest Neighbor, Elastic Net, Logistic Regression, and Support Vector Machines, with the RF model achieving the best classification accuracy at 82.8% (de Lima Lemos et al., 2022). Lalwani et al. introduced more advanced ideas into the data preprocessing problem, using the Gravitational Search Algorithm to select variables and improve the efficiency of the machine learning models (Lalwani et al., 2022). Siddiqui et al. used three variable selection methods to observe the performance of different machine learning models by comparing all variables, separating continuous and discrete variables, and selecting variables based on their importance. They ultimately found that using all variables yielded the most prevalent classification results (Siddiqui et al., 2024).

In addition to focusing on constructing different models to improve classification efficacy, researchers have gradually paid more attention to the interpretability of the models and the relationships between variables. In the field of financial investment, some articles focus on the impact of each variable on volatility risk (Yan and Li, 2024). In the study by AL-Najjar et al., they demonstrated the variable importance of the same traditional machine learning model, C5 Tree, under different methods of variable selection. The results showed that the three main variables affecting the model's effectiveness were the total number of transactions, the total credit card revolving balance, and the change in the number of transactions, providing useful references for credit card managers (AL-Najjar et al., 2022). Other studies discuss variables using more advanced and understandable variable visualization tools, such as the SHAP values method. In Peng et al.'s study, the best model for classification, which combined a genetic algorithm with XGBoost, was used to analyze interpretability. The results of the SHAP values method not only showed the order of importance of different independent variables, but also indicated whether changes in the value of each independent variable had a positive or negative effect on the predicted variable (Peng et al., 2023). In addition to the SHAP values method, other interpretable tools for visualization with similar effects, such as Local Interpretable Model-Agnostic Explanations (LIME), can also be used for analysis (Chang et al., 2024).

So far, machine learning algorithms have primarily captured correlations between variables, and inferring their importance from a correlation perspective, often ignoring the causal relationships among them. In other words, the model's determination of variable importance is frequently based on the strength of the correlation between the dependent and independent variables, which can introduce bias errors in actual predictions (Feuerriegel et al., 2024). To address this issue, this study innovatively uses the R-learner in meta learners to analyze variable importance. This approach combines the fields of machine learning and causal inference (Künzel et al. , 2019). The R-learner employs the control variable method to observe how independent variables affect the probability of credit card customer churn and calculates a more accurate conditional average treatment effect (CATE) by removing bias from the data. This helps quantify the effect of different variables on the probability of credit card customer churn.

3.   Methodology

The aim of this study is to predict whether bank credit card customers will churn using machine learning methods and to utilize the prediction results to construct models that explain the influencing factors of customer churn. This will provide an effective basis for rationalizing customer management before they leave the bank and for effective risk management in the banking industry. Initially, the relevant dataset of bank customer churn needs to be downloaded and preprocessed. Subsequently, the approach is divided into two parts. The first part involves balancing the class distribution of the training set using four sampling techniques: Random Oversampling, Synthetic Minority Oversampling Technique (SMOTE), Borderline-Synthetic Minority Oversampling Technique (Borderline-SMOTE), and Adaptive Synthetic Sampling (ADASYN). Then, six machine learning methods, including RF, Gradient Boosting Decision Tree (GBDT), Extra Tree, AdaBoost, XGBoost, and CatBoost, are used to predict whether the bank will lose credit card customers. The optimal model is selected by comparing the performance of each model, and the important variables influencing customer churn and their effects are analyzed using the SHAP values method. The second part involves using the causal inference method to investigate the causal impact of variables on customer churn based on the optimal prediction model mentioned above. In this part, the R-learner is used for continuous variables in the meta learners of the causal inference method. The framework of this study is outlined in Figure 1.

3.1.   Data preprocessing

Historical data of bank customers downloaded from Kaggle is used as the dataset in this study (Kaggle, 2022). The dataset contains twenty three variables, including customers' fundamental personal details, the credit card level they hold, and their credit card usage, totaling 10,127 samples. The variable labeled "CLIENTNUM" represents the unique identification number assigned by the bank to each customer. Since this variable has no effect on the model construction in this study, it is removed from the analysis. Additionally, the last two variables in the dataset represent the outcomes provided by Kaggle for predicting bank customer churn based on Naive Bayes classifiers. These variables have been excluded during the data processing as this study does not compare this probabilistic classification approach with machine learning methods. The remaining twenty variables, which consist of fourteen quantitative variables and six categorical variables, are all utilized in the experiment in this study to discuss the importance and impact. These variables represent customers' personal information and credit card usage. To build the model, it is necessary to convert the classes of categorical variables into numerical values. These six categorical variables are: "Attrition_Flag", "Gender", "Education_Level", "Marital_Status", "Income_Category", and "Card_Category". For ordered categorical variables, the classes need to be sorted from low to high before conversion. One type of class encountered in the variables "Education_Level", "Marital_Status", and "Income_Category" is "Unknown". There are 3,046 samples that meet the criterion of having at least one of these three variables labeled as "Unknown", constituting 30% of the total sample size. The treatment for "Unknown" in any of these three variables is to change it to the majority class of the respective variable. The details of these six categorical variables are outlined in Table 2.

The variable "Marital_Status" is an unordered categorical variable with more than two distinct values, requiring conversion into dummy variables.

3.2.   Data balancing

The variable "Attrition_Flag" indicates whether a customer has churned or not, serving as the target variable of this study. The dataset contains 1,627 samples in the "Attrited Customer" class. The ratio of these samples to those in the "Existing Customer" class is greater than 5:1, indicating a significant class imbalance, as shown in Figure 2.

If the training set is used by models without preprocessing, it will decrease the ability of the machine learning models to recognize churned customers, leading to overfitting in the results and reducing prediction accuracy (Alam et al., 2020). To avoid these problems and improve the performance of the model, improved oversampling has been adopted to balance the class distribution, which involves increasing the number of samples from the minority class. In this study, the following four sampling algorithms are used.

Random oversampling is a straightforward method that randomly selects and replicates samples from the minority class until the classes are balanced.

SMOTE is an improved algorithm for handling imbalanced data based on random oversampling. The first step involves calculating the distance between each sample from the minority class and identifying the k nearest neighbors of the same class. Then, a specified number of samples are randomly selected from these nearest neighbors. A new synthetic sample is generated between the minority class sample and the selected nearest neighbor, lying on the line connecting these two samples, thereby increasing the number of minority class samples (Akın, 2023).

Borderline-SMOTE is an improved version of the SMOTE algorithm. The key difference is an additional preliminary step: a minority class sample is selected to apply the SMOTE algorithm if most of its k nearest neighbors are from the majority class (Gu et al., 2023).

ADASYN is another common oversampling method similar to SMOTE. The key to this algorithm is calculating the ratio of samples between different classes for each minority class sample and using this ratio distribution to determine the number of synthetic samples generated for each minority class sample (Dube and Verster, 2023).

3.3.   Machine learning evaluation

This study employs six machine learning methods to predict credit card customer churn. In addition to RF and AdaBoost, which were used in the previous research to predict barrier option prices (Li and Yan, 2023), this study utilizes the extra tree algorithm and three other prevalent boosting algorithms: GBDT, XGBoost, and CatBoost. To enhance the performance of these machine learning models, the grid search method is used to tune the hyperparameters of each model and find the optimal set. A complete customer churn prediction model can be constructed by combining one of the sampling techniques mentioned above with one of these six machine learning methods. The prediction will result in four possible cases:

● True positive (TP): The predicted result is positive, and the actual value is also positive.

● True negative (TN): The predicted result is negative, and the actual value is also negative.

● False positive (FP): The predicted result is positive, whereas the actual value is negative.

● False negative (FN): The predicted result is negative, whereas the actual value is positive.

TP and TN indicate the predicted results are consistent with the actual values, and FP and FN indicate the opposite. To evaluate the performance of models in binary classification problems, the following indicators are typically utilized: accuracy, precision, recall, F1 score, and AUC. The formulas for the first four indicators are defined as follows:

The value of AUC is equal to the area under the receiver operating characteristic curve (ROC), which is the curve consisting of recall and false positive rate (FPR). The formula for FPR is as follows:

The closer the values of these five indicators are to 1, the better the model performs.

3.4.   Causal inference

Causal inference is a method that effectively explores and analyzes whether a variable is the main factor affecting the target variable ((Jiang, 2022). In traditional causal inference, the randomized controlled trial is generally considered as a reliable methodology to determine the influence of variables. However, in practice, due to the cost and ethical concerns of experiment, causal inference is often based on collected observational datasets. In Facure's book, which explores the combination of machine learning and causal inference, the authors propose a modeling approach called R-learner for continuous variables in a dataset to assess the causal significance of an independent variable on a dependent variable (Facure M, 2023). In current industrial applications, various code tools are becoming available for researchers to explore the feasibility of machine learning in causal inference problems (Molak A, 2023). To determine whether a variable has a causal relationship with customer churn and to quantify its impact, this study adopts the causal inference method to understand the causal effects among them.

In causality, the cause variable is the treatment denoted as $T$, while the dependent variable is the outcome denoted as $Y$. Additionally, the other independent variables are referred to as characteristic variables, denoted as $X$. For an individual sample $i$, the prediction model predicts the corresponding value $Y_i$ as $Y_i(T = 1|X)$ when the treatment is applied (i.e., $T_i = 1$), and as $Y_i(T = 0|X)$ without the treatment (i.e., $T_i = 0$). Individual treatment effect (ITE) calculates the difference in outcomes for an individual between the scenario with treatment and the scenario without treatment. The formula of ITE is shown as follows:

Considering the overall casual effect between the treatment and the outcome, it needs to measure CATE, which is given by the following equation:

The value of CATE indicates whether there exists a causal relationship between the treatment variable $T$ on the outcome $Y$, and the quantification of its effect.

For binary classification problems, this study uses R-learner to estimate the causality and effect of different continuous treatment variables on the outcome $Y$ respectively, based on the optimal customer churn prediction model. The CATE value in R-learner is obtained by calculating the probability of the same outcome occurring with and without the treatment, which is given by the following equation:

For each $T$, hypothesis testing is applied to CATE$^*$ to determine the casual impact of that treatment variable on $Y$.

4.   Experimental results and analysis

The experimental environment for this study is Python, utilizing toolboxes such as imblearn, sklearn, xgboost, and catboost. Following the data preprocessing steps outlined in Section 3.1, the dataset is divided into a training set and a test set in an 8:2 ratio. The training set data is balanced using four sampling techniques, and the independent variables of all the data are normalized. Subsequently, six machine learning methods are trained on the training set data, resulting in twenty-four complete customer churn prediction models. These trained models are then applied to predict the "Attrition_Flag" values in the test set, and the performance of these models is evaluated using the indicators mentioned in Section 3.3.

The results of the accuracy for the training set, evaluated using six machine learning methods, range from 96.06% to 99.92% under random oversampling, 97.33% to 99.88% under SMOTE, 96.95% to 99.94% under Borderline-SMOTE, and 96.88% to 99.91% under ADASYN. The test set results are as follows.

As shown in Table 2, the values of all indicators for the complete customer churn prediction models are above 0.94 after hyperparameter tuning and training, indicating good performance for each model. The difference in the evaluation results of the prediction models between the training set and the test set is not significant, and there is no instance of complete overfitting. For the random oversampling-XGBoost, SMOTE-XGBoost, Borderline-SMOTE-XGBoost, and ADASYN-XGBoost models, all indicator values exceed 0.97, suggesting that these four models perform better than the others. These four models can be considered the optimal prediction models. Furthermore, the XGBoost model performs better and more stably compared to other machine learning models across different sampling techniques by optimizing both the loss function and regularization term (Guo and Fan, 2024).

The SHAP values method is used to understand the contribution of each variable in the optimal prediction models and its relationship with the target variable better (Wu et al., 2024). When the SHAP value of a variable is positive, it indicates a positive relationship with customer churn, whereas a negative SHAP value indicates a negative relationship. By using the SHAP toolbox in Python, an interpretable explanation of the optimal prediction models is obtained, resulting in corresponding SHAP summary plots. The variables in the SHAP summary plots are sorted from highest to lowest according to their importance in each model, with the top 10 variables selected. The points corresponding to each variable represent their SHAP values.

Based on the four figures above, in any of the optimal prediction models, the variables "Total_Trans_Ct", "Total_Trans_Amt", and "Total_Revolving_Bal" consistently rank in the top three in terms of importance and in the same order. Moreover, the positive or negative relationship between each of these variables and "Attrition_Flag" remains unchanged. These three variables represent the customer's total number of transactions with the bank in the past year, the total amount of transactions, and the total revolving balance of the credit card, respectively. As the total number of transactions increases, the SHAP value decreases, indicating a negative relationship with customer churn. This suggests that customers with a higher number of transactions are less likely to churn. For the second most important variable, the higher the total amount of transactions, the higher the SHAP value. When the total transaction amount is small, the SHAP value can be either positive or negative. There is a significant positive relationship with customer churn only when the total amount exceeds a certain threshold, indicating that customers are more likely to churn when their total transaction amount is above this threshold. For "Total_Revolving_Bal", the greater the total revolving balance of the credit card, the lower the SHAP value, indicating that customers with a larger total credit card revolving balance are less prone to churn compared to those with smaller total credit card revolving balance.

Furthermore, for each optimal prediction model, the variables "Total_Relationship_Count", "Total_Amt_Chng_Q4_Q1", and "Total_Ct_Chng_Q4_Q1" also rank among the top 10 of the SHAP values, consistently appearing in the middle positions. These three variables represent the total number of bank products held by customers, the change in the amount from the fourth quarter to the first quarter, and the change in the number of transactions, respectively. For these three variables, as their values decrease, their SHAP values increase, leading to the predicted result tending closer to 1, which indicates a negative relationship with customer churn. This means that customers who hold a larger number of products or have significant changes in amount and transaction frequency across different quarters are less likely to churn.

For the remaining variables in the SHAP summary plots, although some are important in most optimal prediction models, they pertain to customers' personal information. Therefore, it is not convenient to discuss and classify the attributes of variables such as "Customer_Age", "Marital_Status_0", "Marital_Status_1", and "Education_Level". Additionally, "Months_Inactive_12_mon" for the total number of months inactive in the past year, "Contacts_Count_12_mon" for the total number of contacts in the past year, and "Gender" are only among the top 10 in the random oversampling-XGBoost model. Among these, "Gender" is personal information, while the other two variables are positively related to customer churn. Furthermore, the variable "Avg_Utilization_Ratio" only appears in the ADASYN-XGBoost model, ranking 10th in importance. This variable represents the average utilization rate of bank credit cards, with a lower rate indicating a higher likelihood of customer churn.

The statsmodels toolbox in python is used to analyze the causal effects between variables without personal information of customers and customer churn in the SHAP summary plots. The XGBoost model is selected for estimating customer churn in R-learner due to its superior predictive performance. The experimental results of the causal inference method are listed as follows.

According to Table 3, the p-values for "Total_Revolving_Bal" and "Avg_Utilization_Ratio" are greater than 0.05, indicating no significant difference in customer churn caused by changes in these two treatment variables. Combined with the SHAP values analysis, this suggests that there is only a correlation between each of these two variables and customer churn, not a causal relationship. For the remaining treatment variables, the p-values are all less than 0.05, indicating that changes in each of these variables have causal relationships with customer churn.

Considering the coefficient values in Table 3, the treatment variables "Total_Trans_Ct", "Total_Relationship_Count", "Total_Amt_Chng_Q4_Q1", and "Total_Ct_Chng_Q4_Q1" have negative coefficients, indicating negative causal relationships with customer churn. Among these variables, "Total_Amt_Chng_Q4_Q1" has the largest causal effect, while "Total_Trans_Ct" has the smallest. Conversely, the coefficients of "Total_Trans_Amt", "Months_Inactive_12_mon", and "Contacts_Count_12_mon" are positive, indicating positive causal relationships with customer churn. "Months_Inactive_12_mon" displays the largest causal effect, while "Total_Trans_Amt" displays the smallest. Compared with the results of the SHAP values method, the similarity is that these seven variables have the same direction of impact on customer churn, while the difference lies in the order of the quantified effect values.

5.   Conclusions

In the current research, to enhance the accuracy of prediction results, a combination of sampling techniques and machine learning models was employed to forecast customer churn in banks. A comparative performance analysis indicates that the XGBoost model consistently outperforms other machine learning models, achieving an accuracy of at least 97%, regardless of the sampling techniques used. Furthermore, the SHAP values method was utilized to interpret the optimized prediction models, while R-learner was used to investigate the causal effects of these variables on customer churn. Based on these two methods, the main important variables affecting customer churn, which include the total number and amount of transactions with the bank in the past year, the total number of bank products held by the customer, and the changes in the amount and number of transactions from the fourth quarter to the first quarter, were identified. Additionally, the analysis found that the total credit card revolving balance does not have a significant causal relationship with customer churn, but there is a strong correlation. The research findings provide valuable recommendations for bank managers to improve customer management strategies.

Due to the limited number of minority class samples in the dataset, the experiment requires sampling techniques to generate synthetic samples. This approach enables the prediction model to more accurately identify the categories of samples. Furthermore, excluding the variables belonging to customers' personal information, the other variables consist of cross-sectional data. They are typically utilized in the analysis that emphasize the differences among individual samples, rather than focusing on changes within a sample over time. Therefore, if additional samples with more extensive variables, such as time series data, are available the feasibility of the model's predictions, so that a better comparison of the interaction between the SHAP values method and the casual inference method in the further research.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this study.

Acknowledgments

The authors have received no financial assistance from any source in the preparation of this study.

Conflict of interest

All authors declare no conflicts of interest in this study.