A screened predictive model for esophageal squamous cell carcinoma based on salivary flora data

1.   Introduction

ESCC is a malignant epithelial tumor with squamous cell differentiation and is a type of EC. ESCC mainly occurs in Asian countries such as China ^[1], the incidence and the mortality rates of ESCC are among the highest for malignant tumors ^[2]. In China, more than 90% of EC cases are ESCC ^[3], accounting for approximately half of all ESCC cases worldwide ^[4]. With the improvement of medical treatment in recent years, the morbidity rate and death rate of EC have decreased. However, due to the lack of early disease screening methods and the fact that patients with EC do not show obvious symptoms until the middle to late stages of the disease, the survival rate of patients within 5 years is only 13–18% ^[5]. Therefore, it is essential to prevent the disease early and to develop a screening tool for ESCC.

Many factors contribute to the development of ESCC ^[6], factors such as alcohol consumption, smoking, diet and nutritional deficiencies are the main causes of ESCC ^[7]. Moreover, poor oral health is also a risk element for ESCC lesions, and this finding has been reported in China, Japan, India and Latin America ^[8,9,10]. In addition, there is an association between oral microbiota and systemic diseases ^[11,12,13]. Since the composition of the oral microbiota is similar to that of the esophageal microbiota, alterations in the oral microbiome may lead to alterations in the esophageal microbiome ^[14], which in turn may cause EC ^[15,16]. Some researchers have indicated that the oral microbiota of patients with ESCC differs from that of the controls ^{[11,17,18,19,20,21,22,23,24]}. Wang et al. and Li et al. concluded that the relative abundance of Bacteroides on ESCC was significantly lower than that of the control ^[18,22]. Furthermore, a study from the United States showed that an increase in Prevotella oral taxon 306 belonging to Bacteroides was associated with a low risk of ESCC ^[24]. For Proteobacteria, a study by Chen et al. showed a reduction in the abundance of Neisseria in ESCC patients ^[17]. Several other studies on the relationship between oral flora and ESCC risk also showed a significant decline in Proteobacteria among ESCC patients compared to controls ^{[18,22,25,26]}. Zhao et al. found that Veillonella, which belongs to Firmicutes, was enriched in EC patients ^[21]. Moreover, the finding in Lu's study ^[27] that the risk of suffering from ESCC was associated with an increased relative abundance of Fusobacteria and Actinobacteria in saliva samples. Therefore, it is necessary to explore the association between the relative abundance of oral flora and ESCC.

Machine learning is the study of how computers can simulate the implementation of human learning behaviors and thus acquire new knowledge and skills from mountains of data. Machine learning (ML) can help intelligent systems make a variety of judgments based on data. Common ML-supervised learning algorithms include RF, GMM, KNN, LR, SVM and XGBoost. These algorithms are commonly used for classification problems and regression problems. Nowadays, machine learning is widely used in biomedicine as a mainstream data processing method, and this discipline has enabled risk prediction for a variety of diseases such as cancer ^[28]. Furthermore, ML models were utilized as early as 2005 for survival prediction in patients with esophageal cancer. Mofidi et al. used clinical information, body mass index and pathology datasets of patients with EC to develop models to predict survival in patients with EC and surgical resection of the esophagogastric junction ^[29]. Hayashida et al. trained models to predict the efficacy of chemotherapy and radiation therapy in patients with esophageal cancer from protein profile data obtained from serum samples ^[30]. However, the development of predictive models for EC is lagging compared to survival models, it is still an open question of how to use oral flora data to build ML models for prediction and early screening of ESCC.

Saliva samples are easy to collect and widely used to detect and classify microorganisms ^[31]. Meanwhile, considering that the composition of human flora is susceptible to change due to multiple external factors and human activities, such as residence location and dietary habits ^[32], data from more sources would help to explore deeply the relationship between oral flora and ESCC. In this study, RF, GMM, KNN, LR, SVM and genetic algorithm-based optimization XGBoost (GA-XGBoost), will be built to distinguish healthy controls from ESCC patients based on the relative abundance data of phylum level bacteria in saliva such as Bacteroides, Firmicutes, Proteobacteria, Fusobacteria and Actinobacteria, which associated with ESCC ^{[17,18,19,22,25,26,27,33,34]}. The ML models also provide early warning for patients at risk of ESCC, allowing sufficient time for intervention and treatment to improve survival.

2.   Materials and methods

2.1.   Data collection and processing

The data used in this study are from 8 articles published between 2015 and 2021 that explored the relationship between salivary microbiota and ESCC ^{[17,18,19,22,25,26,27,33,34]}. This study was conducted under approved guidelines as all included literature is publicly available.

To analyze the relationship between ESCC and salivary microflora, data based on the mean and standard deviation of relative abundance of Bacteroides, Firmicutes, Proteobacteria, Fusobacteria and Actinobacteria reported in each article for ESCC patients and healthy controls were generated. For the relative abundance data presented in the form of graphs, values were extracted through the Web Plot Digitizer (version 4.6) and the data were represented as box plots, with the mean and standard deviation calculated from the quartiles of the box plots ^[35]. If the relative abundance data of a particular bacterium were missing in one article, the average value of that bacterium in the rest of the articles was used to fill in the data. For example, the relative abundance data of Fusobacteria are missing in the study of Wan et al., and the average of the relative abundance of Fusobacteria in the remaining 7 articles is calculated and that value is used as the relative abundance measure of Fusobacteria in the study of Wan et al. ^[35].

A total of 8000 sets of data from 8 sources ^{[17,18,19,22,25,26,27,33,34]} were generated, which included the relative abundance data of Bacteroides, Firmicutes, Proteobacteria, Fusobacteria and Actinobacteria in 4000 ESCC patients and 4000 healthy controls (Table 1).

2.2.   Machine learning methods

The analyses were conducted with Python programming language (version 3.9) and the "scikit-learn" library. Using a series of ML methods: RF, GMM, KNN, LR, SVM and XGBoost, a binary classification model was constructed according to oral salivary flora (0: non-ESCC patients, 1: ESCC patients) to predict ESCC ^[36].

2.2.1.   K-fold cross-validation

K-Fold cross-validation ^[37] is a common method for parameter optimization and model selection in ML. It randomly divides the dataset into $k$ disjoint subsets of the same size, selects one subset each time as the testing set and the remaining $k-1$ subsets as the training set, then repeats $k$ times to obtain $k$ models. Calculate the test error of each model and use the average of $k$ test errors as the performance metric of the model. The performance index of the model $E$ can be calculated as follows:

where $E_i$ represents the test error of the $i$-th model.

2.2.2.   Random forest (RF)

Random forest ^[38] is a classifier that uses multiple decision trees to train and predict samples. In this model, a training set is formed by sampling $N$ times from $N$ samples in the way of resampling, and the unsampled samples are used to predict and evaluate its error. The probability $P$ that each sample is not drawn is calculated as follows:

For each node of the decision tree, $m$ features are randomly selected and their optimal split is calculated based on these features by the Gini index. The Gini index $G(M)$ can be calculated as follows:

here $M$ is the training subset and $p_k$ is the probability of a sample belonging to the $k$-th category. RF can handle both classification and numerical features, and the model is also relatively resistant to over-fitting, making false predictions only when more than half of the base classifiers are in error. However, due to its inherent complexity, it requires more time to train them than other similar algorithms.

2.2.3.   Gaussian mixture models (GMM)

The GMM ^[39] is a widely used clustering algorithm that uses a Gaussian distribution as a parametric model. GMM can be regarded as a model consisting of a combination of $K$ Gaussian models. The data $x = (x_1, x_2, \ldots, x_n)$ is generated from several standard Gaussian distributions, then the probability density $P(x)$ of the Gaussian distribution is computed as follows:

where $\mu$ is the expectation of data and $\Sigma$ is the covariance. The Gaussian mixture model is defined by the formula:

where the mean $\mu_i$ and variance $\Sigma_i$ are the parameters to be estimated, $\pi_i$ is the probability of generating data for each sub-model. GMM can be used not only for clustering but also for probability density estimation. However, estimating covariance can be difficult when there are not enough points for each mixture model.

2.2.4.   K-nearest neighbor (KNN)

KNN ^[40] is a basic classification and regression method. In the classification problem, each data in the sample set containing $n$ samples $x_i = (x_{i1}, x_{i2}, \ldots, x_{in})$ have a corresponding relationship with their classification. After inputting a new sample without labels, each feature of the new data is compared with the corresponding feature of the data in the sample set, the algorithm extracts the classification labels of the $k$ most similar data of the sample and finally selects the classification label with the most occurrences among the $k$ most similar data as the label of the new sample. The model generally selects the nearest neighbors by calculating the distance from the test data to each of the training data. The distance $d$ is commonly calculated by the formula:

KNN is a mature theory with simple ideas, and can be used for both classification and regression. However, the prediction bias is relatively high when the samples are not balanced.

2.2.5.   Logistic regression (LR)

LR ^[41] is a ML method used to solve binary classification problems and is also a generalized linear model. It maps the input set of data $x = {(x}_1, x_2, \ldots, x_n)$ with $n$ features to a number between 0 and 1 by the sigmoid function. When the function value is greater than 0.5, the classification result is judged as 1 (ESCC patients), otherwise, the classification result is 0 (non-ESCC patients). The predicted result $y(x)$ is computed as followed:

here $T$ means transpose, $w$ and $b$ are parameters to be evaluated. LR is computationally small and has a low storage footprint, which can be used in big data scenarios and is especially suitable for classification problems. However, it is essentially a linear classifier, so it does not handle the case of correlation between features well.

2.2.6.   Support vector machine (SVM)

SVM is a kind of ML model that classifies data in a supervised learning manner, which uses a hinge loss function to compute empirical risk and adds a regularization term to the solution system to optimize structural risk. SVM is one of the common kernel learning methods and the basic concept of SVM is to find the separating hyperplane that can correctly partition the training data set with maximum geometric separation. SVM has high accuracy and provides a good theoretical guarantee to avoid over-fitting. However, the model is memory-consuming and difficult to interpret, and the efficiency is not great when there are many observed samples.

2.2.7.   Extreme Gradient Boosting (XGBoost)

XGBoost is an efficient gradient-boosting decision tree algorithm that is composed of multiple classification regression trees, so the algorithm can process problems such as classification regression. As a forward addition model, the core of the XGBoost algorithm is to integrate multiple weak learners into one strong learner by certain methods. The model uses multiple trees to make decisions together. Each tree is the result of the target before and all the trees of the difference in value between predicted results, and the final result is obtained by combining the results of all the trees. Given a dataset containing $n$ samples, there is a corresponding true result ${\hat{y}}_i$ for any sample $x_i(i\epsilon\left(1, 2, \ldots, n\right))$. The predicted probability ${\hat{y}}_i$ can be calculated as follows:

where $K$ denotes the number of decision trees, $f_k$ is the prediction score of a single decision tree and $\mathcal{F}$ is the space composed of all trees. The objective function $obj$ of XGBoost can be expressed as follows:

here $l$ is the loss function and $\Omega$ is the penalty term. The expression for the penalty term is:

where $T$ indicates the number of leaves of a single tree, $\omega_i$ is the score of the leaf node $i$, $\gamma$ and $\lambda$ are hyperparameters. XGBoost has very high accuracy as well as greater flexibility, but the model still requires traversal of the dataset during node splitting.

2.2.8.   Genetic algorithms (GA)

The GA is a computational model of the biological evolutionary process based on the Darwinian biological evolutionary theory of natural selection and genetic mechanism, which searches for the optimal solution by simulating the natural evolutionary process. The algorithm targets all individuals in the population and uses randomization techniques to perform an efficient search of a parameter space being encoded. In this case, selection, crossover and mutation constitute the genetic operations of the GA. After the initial population is generated, the evolution of generation by generation produces increasingly better approximate solutions according to the principles of survival of the fittest. In each generation, individuals are selected for their fitness in the problem domain, and new populations are generated by combinatorial crossover and mutation with the help of natural genetic operators.

2.2.9.   XGBoost based on genetic algorithm optimization (GA-XGBoost)

Genetic algorithms are often used for model parameter tuning, which can make the model have better fitting or prediction ability. The most important step in the XGBoost model is model tuning, so the genetic algorithm can be combined with XGBoost, using the genetic algorithm to adjust the parameters of the XGBoost model, to obtain the best model.

2.3.   Model performance evaluation

For a binary classification model, there are four classification results after the classifier divides the instances into positive and negative classes:

True Positive ($TP$): True positive example, an instance is positive and is also determined to be positive.

True Negative ($TN$): The correct negative case, where an instance is a false category and is also judged to be a false category.

False Negative ($FN$): False negative example, an instance is a positive category but is judged as a false category.

False Positive ($FP$): Wrong positive example, which is a false category but is judged to be a positive category.

Accuracy is the degree to which a thing is expressed or described correctly. The formula for calculating the Accuracy is:

Precision is the accuracy of the prediction of positive cases, which is the proportion of the sample predicted as positive cases that are predicted correctly, and it is used to assess the accuracy of predicting positive cases. The formula for calculating the Precision is:

Recall is the proportion of positive cases that are predicted correctly to the total sample of actual positive cases in the sample of actual positive cases. The formula for calculating the Recall is:

Precision and Recall are a pair of contradictory metrics. In general, Recall values seem to be low when Precision is high, while Recall values tend to be high when Precision is low. In order to be able to consider these two metrics together, F1-score is proposed, which is a weighted summed average of Precision and Recall, calculated by the formula:

The ROC curve describes the variation of classifier performance with the classifier threshold ^[42], and each point on the curve reflects the perceptibility to the same signal stimulus. This curve is a curve plotted with the true positive rate as the vertical coordinate and the false positive rate as the horizontal coordinate. When comparing the performance of two or more models, it is possible to visually identify the strengths and weaknesses of the models. The ROC curve of the model closer to the top left corner means that the model is more accurate. The true positive rate $TDR$ was determined as follows:

The expression for the false positive rate $FDR$ is:

The AUC is considered as the area under the ROC curve, the larger area under the curve, the better the performance of the corresponding model. AUC values range from 0 to 1 and are generally between 0.5 and 1 ^[42].

2.4.   Experimental design

The dataset ($N$ = 4000 ESCC patients and $N$ = 4000 healthy controls) compares six machine learning models (RF, GNN, KNN, LR, SVM and XGBoost) with default parameters in the "scikit-learn" library for Python utilizing a five-fold cross-validation method: Randomly divide the dataset into five groups, four for training the model and the remaining group for testing. This process is repeated five times so that each group is used as a testing set once. The average performance of all five groups is considered the final performance of the model. Repeat this process five times for each ML algorithm and show the mean value of each model after five training sessions. The model with the highest accuracy (XGBoost) was selected as the final prediction model.

This dataset was divided into a training set (80%) and a testing set (20%). The parameters of the model with the highest accuracy are optimized on the training set using GA, and the performance metrics (Accuracy, Precision, recall, F1-score, ROC and AUC) of the parameter-optimized model are evaluated on the testing set.

2.5.   External verification

Since oral flora presents some variation due to multiple factors ^[32], this work verified the ability of the model by generating 1000 sets of data ($N$ = 500 ESCC patients and $N$ = 500 controls) with means and standard deviations of the relative abundance of Bacteroides, Firmicutes, Proteobacteria, Fusobacteria and Actinobacteria which reported in Wei ^[33]. Table 2 illustrates the data characteristics of the external validation queue.

3.   Results

3.1.   Statistical analysis of salivary flora abundance data

We included data on the relative abundance of salivary flora from 4000 ESCC patients versus 4000 healthy controls, with 6400 data sets in the training set and 1600 data sets in the testing set. In the training set, the number of ESCC patients ($N$ = 3177) was less than the number of controls ($N$ = 3223). Conversely, in the testing set, there were more data from the ESCC group ($N$ = 823) than from the healthy controls ($N$ = 777). Compared to the controls, the oral saliva of the ESCC patients showed a lower number of floras belonging to Proteobacteria, Fusobacteria and Actinobacteria. Meanwhile, the elevated relative abundance of Bacteroides and Firmicutes, may be associated with ESCC (Table 3).

3.2.   Model construction and evaluation

The data were cross-validated in five folds with the default hyperparameters of ML models and compared the predictive power of six models (RF, GMM, KNN, LR, SVM and XGBoost). According to the average accuracy, XGBoost was most effective in predicting the risk of a sample suffering from ESCC and correctly predicted 73.50% of the samples. The performance of the RF is the second highest after the performance of XGBoost, with an accuracy of 73.34%. The accuracy of the other models is lower compared to the accuracy of the first two models, including KNN (Accuracy = 70.28%), SVM (Accuracy = 68.45%), GMM (Accuracy = 58.16%) and LR (Accuracy = 57.36%) have relatively poor performance (Table 4).

The parameters in the XGBoost model were optimized in the training set with GA, and the parameters n_estimators were cross-validated to obtain the best value of 82. The remaining parameters were iterated through 50 iterations of GA to obtain the parameter combinations that optimize the performance of the XGBoost model. Table 5 shows the parameter combinations and model capabilities of the GA-XGBoost model and other models (XGBoost, RF, GMM, KNN, LR and SVM) used on the testing set.

The accuracy of GMM, LR and SVM in predicting whether a sample suffered from ESCC through the relative abundance data of salivary flora was less than 80%. Compared to KNN (Accuracy = 85.44%), XGBoost (Accuracy = 88.19%) and RF (Accuracy = 89.56%), GA-XGBoost (Accuracy = 89.88%) has a better accuracy in predicting ESCC. Precision and Recall, as a pair of contradictory evaluation metrics, both show the best performance in the GA-XGBoost model (Precision = 89.43%, Recall = 90.75%), followed closely by RF (Precision = 89.06%, Recall = 90.49%), XGBoost (Precision = 87.85%, Recall = 89.04%) and KNN(Precision = 82.87%, Recall = 88.11%). The Precision of SVM is not low, but the Recall is below 80%, approximately 78.33%. F1-score is an evaluation metric that balances both Precision and Recall. So for F1-score, GA-XGBoost is also the most effective model among the seven models (F1-score = 90.09%). However, GMM and LR performed negatively in the four evaluation metrics of accuracy, Precision, Recall and F1-score. Except GMM, which had a Precision of about 70.35%, none of the other metrics exceeded 70%. The performance of these models was further evaluated using the AUC of ROC, and GA-XGBoost had similar AUC values to RF as well as XGBoost, both at 0.97, followed by KNN (AUC = 0.94), SVM (AUC = 0.89), GMM (AUC = 0.67) and LR (AUC = 0.62).

Based on a comprehensive evaluation of five evaluation metrics, the GA-XGBoost model showed the best performance in predicting ESCC. Therefore, the XGBoost model optimized based on GA is more effective in exploring the relationship between oral flora and ESCC than other traditional ML models.

3.3.   External validation of XGBoost optimized by genetic algorithm

Since GA-XGBoost has the best performance among all models, in order to verify the generalizability of the model, the XGBoost model with the same parameters are used for validation in an external cohort. The external cohort provided 1000 samples to validate the model. GA-XGBoost revealed a better prediction performance in the external cohort. On the one hand, based on the results of the confusion matrix (Figure 1), the model achieves 70.60% Accuracy, 46.00% Precision, 90.55% Recall and 61.01% F1-score (Table 6). On the other hand, the AUC value of the model (AUC = 0.88) achieved similarly positive results (Figure 2).

3.4.   Importance ranking of predictor variables

The ranking of feature importance calculated from the XGBoost model after optimizing the parameters showed that Bacteroides was the most important predictor. During the construction of the decision tree in XGBoost, the number of key decisions made conditional on Bacteroides was 319, accounting for 24.67% of the total number of decisions. The following predictor was Actinobacteria (21.73%). Although the importance index values of Firmicutes, Proteobacteria and Fusobacteria were less than that of Bacteroides and Actinobacteria, they all ranged from 15 to 20% (Figure 3).

4.   Discussion

Early diagnosis and screening of cancer is an important method to improve the cure rate of cancer, prolong the survival time of patients and reduce the financial and emotional burden of patients. In the past decade, artificial intelligence technologies and ML approaches have been widely used in the early cancer screening, clinical aid diagnosis and mechanistic research. However, EC is not easily detected in the early stage, and ML research in the field of EC is lagging behind compared with other cancers, so building ML models is crucial in EC research. In this study, six ML models were applied to explore the relationship between the relative abundance of oral salivary flora and ESCC. The results indicated that the GA-XGBoost model observed higher performance in predicting whether a sample was suffering from ESCC or not. This is because the XGBoost model, as an integrated algorithm, has an advantage over other models by adding additional canonical terms to avoid model overfitting in the construction of decision trees ^[43]. According to our knowledge, this is the first study to use the ML methods approach to model and predict the presence of ESCC by the relative abundance of flora at the salivary phylum level.

Previous studies have examined oral microbiota and periodontal characteristics as clinicopathological factors in patients with EC ^[20,44]. Moghtadaei et al. constructed a risk prediction model based on common risk factors for EC such as age, residence, smoking history and oral health, which obtained an accuracy of nearly 90% and surpassed the traditional LR model ^[20]. Kawasaki et al. confirmed that the prevalence of T. forsythia and S. anginosus in dental plaque as well as the prevalence of A. actinomycetemcomitans in saliva and alcohol consumption habits were associated with a high risk of EC ^[20]. Then he constructed a multiple LR model with the relative abundance of S. anginosusd in saliva and sociological factors such as age, body mass index, smoking history and alcohol consumption habits as variables, which had an AUC value of 0.82. However, these models are often suitable for large sample surveys and are not applicable to individual clinical diagnosis because factors such as patients' residence, smoking history and drinking habits change according to people's habits, making it impossible to obtain comprehensive and accurate data.

Saliva is not only easy to extract samples, but also effective in making predictions and judgments about whether an individual case has ESCC or not. Based on this property of saliva, a GA-XGBoost model with 89.88% Accuracy, 89.43% Precision, 90.75% Recall, 90.09% F1-score and AUC = 0.97 was constructed by using oral saliva flora in this study. Even though the performance of the GA-XGBoost model was slightly reduced in the external validation set, the prediction performance of the model still performed positively (Accuracy = 70.60%, Precision = 46.00%, Recall = 90.55%, F1-score = 61.01%, AUC = 0.88). It is noteworthy that the recall of the model in the external queue reaches 90.55%, but the precision is only 46.00%. Since recall and precision affect each other and are mutually constrained, a high value for both cannot generally be achieved in realistic situations. In the field of disease prediction, it is acceptable to misclassify a disease, but not to undiagnose a disease. Therefore, Recall's value should be more meaningful. These results show the ability of salivary flora and GA-XGBoost models to predict ESCC. Although salivary flora is influenced by a relatively large number of factors, the model is capable of effectively predicting the presence or absence of ESCC.

In this study, we considered the importance of five phylum-level bacteria: Bacteroides, Firmicutes, Proteobacteria, Fusobacteria and Actinobacteria for ESCC. Our model suggests that Bacteroides and Actinobacteria are the two most important influencing elements, which have a powerful impact on the prediction of ESCC. A study from China and a study from the United States reported that alterations in Prevotella belonging to Bacteroides may be a potential predictor of EC ^[21,24]. Furthermore, Kawasaki et al. suggested that Actinobacteria could be used for the diagnostic evaluation of esophageal cancer even though it is a rarely detected periodontal disease bacterium ^[20]. Therefore, the relationship between Bacteroides, Actinobacteria and ESCC can be further explored.

All the data in this study were sourced from ESCC patients with healthy controls in China. Since the flora in the human body is susceptible to multiple factors, the capabilities and parameters of the model may be altered when the model is in application to other regions or environments. In addition, the present study only considered the association between oral microorganisms and ESCC. In order to further improve the performance of the model, sociological factors such as region, age, gender and drinking habits could be included in the model. Regardless of these limitations of the current study, modeling the relationship between oral flora and ESCC can provide further insight into the role played by oral flora in ESCC, leading to non-invasive methods for the prevention and diagnosis of ESCC.

5.   Conclusions

In this study, following comparison and screening, a GA-XGBoost model with parameter optimization of the XGBoost algorithm using the global search capability of the GA was proposed to establish a disease prediction model for early screening of ESCC by the relative abundance of oral salivary flora. The parameters of the XGboost model were not easily determined to affect the performance of the model, and the accuracy of the algorithm was improved while avoiding the occurrence of overfitting. The experimental results show that the genetic algorithm improves the performance of the XGBoost model, and the GA-XGBoost model has better judgment than the traditional machine learning algorithm. The study provides a new biomarker for the diagnosis of ESCC by the relative abundance of Bacteroides and Actinobacteria, which also provides a new research direction to investigate the association between oral flora and ESCC.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Conflict of interest

The authors declare there is no conflict of interest.