Gradient Lagrangian systems and semilinear PDE

1.   Introduction

With the popularity of big data, the amount of information that people can obtain increases exponentially. Therefore, it is particularly important to use a large amount of data to analyze and get useful results. As a means of processing big data, information fusion can effectively integrate all kinds of information and guide people's production and life. Evidence theory ^[1,2], as an influential method for information fusion, can effectively deal with uncertain information ^[3,4]. This has garnered significant attention and study from scholars ^[5,6,7,8], leading to its application in various domains such as pattern recognition ^[9,10,11], risk assessment ^{[12,13,14,15]}, and multi-attribute decision making ^[16,17,18].

Evidence theory provides a fusion rule, namely the Dempster's combination rule. It is employed to combine multiple different sources of evidence into a comprehensive body of evidence. The purpose of this rule is to effectively handle and merge various pieces of evidence in situations characterized by uncertainty and incomplete information. However, classic Dempster's combination rule requires that evidences should be independent of each other ^[19]. This assumption limits the application scope of evidence theory to some extent, because dependence is universal in practice. For example, in the decision-making process, experts will discuss and exchange opinions with each other, so the conclusions they draw will synthesize their opinions and be dependent. If the evidence intersects is not taken into account, the influence of dependence among factors on the results will be calculated repeatedly in the process of evidence combination, leading to incorrect results. In order to solve this problem, many scholars put forward different methods. These methods fall into two main categories: improving the combination rule and modifying the original evidence.

The first type of method is to improve the combination rule. Since Dempster's rule requires evidences to be independent, an evidence fusion method without this constraint can be constructed ^[20,21]. Chebbah et al. ^[21] suggested a method to quantify the degree of independence between evidence sources, which helps to select the most appropriate set of combinatorial rules to fuse evidence information from different sources. However, the method requires a sophisticated process of clustering, matching clusters, and quantifying independence, which might be computationally intensive and complex. Destercke et al. ^[22] made a more general definition of a rule in possibility theory and proposed the fusion rule, so that it can be used in the fusion of dependent evidence, however, Cattaneo ^[23] argued that this rule does not satisfy the basic evidence theory. Fu et al. ^[24] developed a method for fusing dependent interval-valued reliability functions. However, a potential limitation of this approach is its inherent complexity in both implementation and interpretation, particularly in scenarios characterized by significant interdependence among attributes or in dynamic decision-making environments.

The second type of method is to modify the original evidence, the main idea of which is to reduce the dependence among information sources and make the information sources approximately independent by discounting the dependent information, so as to reduce the repeated calculation in the process of fusion ^[20]. Dempster's combination rule has many advantages, such as satisfying the commutative law, associative law, and other mathematical characteristics, so the second type of method is attracting more and more attention. Su et al. ^[20] proposed an improved method for combining dependent evidence bodies, which takes the significance of the common information sources into consideration. However, it is constrained to scenarios involving only two sources. Extending this approach to incorporate multiple sources remains an unresolved issue. Su et al. ^[19] proposed a method to deal with dependent evidence at the system level, which can grasp internal and external dependence. However, this approach is somewhat subjective. Shi et al. ^[25] used the rank correlation coefficient to generate the discount coefficient of dependent information. Su et al. ^[26] proposed an evidence fusion model based on mutual information to discount BBA. However, these methods cannot effectively deal with information redundancy. Kong et al. ^[27] developed a clinical decision support system (CDSS). Yao et al. ^[28] proposed a maximum likelihood evidential reasoning (MAKER) framework. These models build evidence bases from clinical historical data. The discount coefficient of evidence is derived from the training model. However, this model requires huge amounts of historical data, and the effect can be affected by the size and quality of the data set.

Principal component analysis (PCA) is a widely used data dimensionality reduction algorithm. The basic idea of PCA is to retain the most important features of the closely dependent high-dimensional data and turn it into the approximately mutually independent low-dimensional data, that is, the principal components, so that a small number of comprehensive indicators can be used to represent the most important information existing in the original data. PCA is widely used in computer science ^[29,30,31], engineering ^[32,33,34], and other fields.

Therefore, following the idea of the second type of method, this paper proposes a dependent evidence fusion model based on PCA. Variables in the original data set may be dependent. Through linear combinations of these variables, PCA creates a new set of variables (the principal components) that are orthogonal to each other, meaning their dependence is greatly reduced or eliminated. Although PCA cannot guarantee complete independence (as this requires statistical independence and PCA only eliminates linear correlation), it significantly reduces the depenence between data features. Through coordinate transformation, PCA can effectively deal with the redundancy of information. PCA converts dependent data into approximately independent principal components, which means the evidence processed by PCA can approximately conform to the assumption of Dempster's combination rule.

This paper is organized as follows: Section 2 mainly introduces PCA and related concepts of evidence theory. Section 3 presents the steps of the proposed method. In Section 4, the effectiveness and superiority of this method are demonstrated through several experiments. Section 5 summarizes the method.

2.   Preliminaries

2.1.   D-S evidence theory ^[1,2]

2.1.1.   Discernment frame

Define the discernment frame $\Phi$ as the set of all possible and independent values of the variable $x$. Let the number of elements in $\Phi$ be $\mu$, then the power set $P(\Phi)$ of $\Phi$ has $2^{\mu }$ elements, each of which corresponds to a proposition about the value of $x$.

2.1.2.   Basic belief assignment (BBA)

Let any element of $P(\Phi)$ correspond to a number whose value ranges $\left [0, 1 \right]$. If the mapping is satisfied:

then $m$ is the basic belief assignment (BBA) on $\Phi$. $m(N)$ is a probability function that represents the degree to which the evidence supports proposition $N$ to be true.

The evidence theory supports experts to give the confidence $\beta$ of their judgment, the BBA $m'$ discounted by confidence $\beta$ is:

2.1.3.   Dempster combination rule

In order to make the BBA of experts work together, it is necessary to establish the corresponding combination rules. Let $m_{1}, m_{2}, \dots, m_{h}$ be $h$ independent BBAs in $\Phi$, corresponding to the propositions $N_{1}, N_{2}, \dots, N_{h}$. Then the result of the combination is their orthogonal sum as follows:

and the BBA of the fused proposition $Y$ is

where $C$ is a normalized parameter reflecting the conflict degree between the evidence.

2.1.4.   Pignistic probability transformation (PPT)

Let $m$ be a BBA on $\Theta$. Its corresponding pignistic probability function $Bet{P_m}: \Phi\longrightarrow [0, 1]$ is defined as

where $\left| N \right|$ is the cardinality of subset $N$.

2.1.5.   Application of evidence theory

The general usage process of evidence theory in the decision support field is as follows: experts evaluate a discernment frame, and based on the current context, prior knowledge, and experience, they provide judgments on propositions within the discernment frame. Evidence theory supports experts in assigning basic belief assignments (BBA) and confidence levels to any combination of propositions within the frame. Through Dempster's combination rule, the opinions of different experts are integrated to obtain a comprehensive proposition (the fused BBA) that reflects the consensus of all experts.

Take medical diagnosis as an example to illustrate the application process of evidence theory. In medical diagnosis, the discernment frame consists of $n$ possible diseases. Experts provide judgments and confidence levels on which disease it could be based on symptoms and prior knowledge. For instance, Expert 1 might offer an opinion on Disease A with a certain confidence level (BBA 1), while Expert 2 might provide an opinion on either Disease A or B with a different confidence level (BBA 2). Evidence theory integrates the opinions of these experts to arrive at a final judgment opinion (fused BBA). The final opinion reveals the disease and its probability level, incorporating the views of both experts.

2.2.   Principal component analysis (PCA) ^[35]

The specific steps of principal component analysis are as follows:

Step 1. Set the total number of samples in a data set as $n$ and the number of attributes as $p$, then matrix $X = \left [X_{ij} \right] (i\in \left [1, n \right], j = \left [1, p \right])$ can be obtained from the original data of the samples, where $X_{ij}$ is the observed value of the $j$th attribute of the $i$th sample.

Step 2. Standardization of raw data. In general, the contents described by different attributes of data are different, they represent different physical meanings, and have different units of measurement. If this point is ignored and the principal component is calculated directly, it will lead to incorrect results. Therefore, it is necessary to standardize the raw data so that the influence of each attribute is at the same level. The standardized data $Z = \left [Z_{ij} \right] (i\in \left [1, n \right], j = \left [1, p \right])$ is defined as follows.

Step 3. Compute the correlation coefficient matrix $R$. The correlation coefficient represents the dependence degree between standardized data, and the smaller the value, the smaller the degree. There is little information overlap between variables with low dependence degree, otherwise, the information of variables will overlap, resulting in redundant information. Redundant information affects the objectivity of decision making. In order to identify the dependence of variables, it is necessary to construct matrix $R = \left [R_{ij} \right] _{p\times p}$.

The matrix $R$ is calculated according to the standardized data, where $R_{ij}$ reflects the dependence degree between index $Z_{i}$ and $Z_{j}$, and the expression is as follows:

where $\bar{Z_{i}}, \bar{Z_{j}}$ is the sample average of $Z_{i}$ and $Z_{j}$.

Step 4. Compute the eigenvalues $\lambda_{1}\ge \lambda_{2}\ge\cdots \ge\lambda_{K}\ge0$ and the eigenvector $V_{j} = (V_{1j}, V_{2j}, \cdots, V_{pj})^{T}$ of the correlation coefficient matrix $R$ according to Eq. (2.10).

The eigenvector represents the dependence among the new $K$-dimensional data and the original $N$-dimensional data, and the larger the eigenvalue, the more representative the new data is of the corresponding original data. The eigenvalue is the variance of each principal component and represents the contribution degree of each principal component to the final result. $V_{ij}$ is the element in row $i$ and column $j$ of the eigenvector matrix.

Step 5. Let $X_{j} = (X_{1j}, X_{2j}, \cdots, X_{nj})^{T}$, then the original data matrix is $X = \left [X_{1}, X_{2}, \cdots, X_{P} \right]$. The principal component values are expressed as follows:

3.   The proposed method

In order to fuse the dependence information accurately, this part proposes a method of dependence evidence fusion based on PCA.

Step 1. Acquire raw data

Acquire the original data matrix $X = \left [X_{ij} \right]_{n\times p}$ to serve as the input data for PCA, as folows:

Step 2. Obtain principal components through PCA

There are common (redundant) parts among dependent information, which will be calculated repeatedly in the process of information fusion, resulting in unreasonable fusion results. To address this issue, this paper employs principal component analysis in the fusion of evidence. The core principle of PCA involves maximizing the variance of data points through coordinate rotation and transformation. This process enhances the most significant features (principal components) of the data while reducing dimensionality. In this way, PCA is able to identify and prioritize those components that capture the greatest variance in the data, thereby preserving critical information. Moreover, PCA eliminates redundant features from the data by selecting the principal components that best describe the data's variance. This means it selects the features that best explain the variance of the data while disregarding those features that are highly correlated with the selected features. Therefore, this paper use the characteristics of PCA to handle dependence among variables.

Through the basic steps of PCA (see Section 2.2), the original data set is transformed into independent principal components.

Step 3. Build BBAs of principal components

Build the BBA. This paper adopts the method of Xu et al. ^[36] to generate the BBA based on the data set of independent principal components after PCA processing.

Step 4. Fuse the BBA and draw a conclusion

The results are based on the fusion of the BBA. Eq. (2.5) and Eq. (2.6) are used to obtain the final fusion result.

4.   Case study

The iris flower data set was first measured by Edgar Anderson and later used by R. Fisher in his 1936 paper ^[37] as an example of classification methods in statistics. Since then, this data set has been studied by many scholars, especially in the field of machine learning. The attributes of iris are dependent and can be seen as information sources for building BBAs. The species of iris can be identified through the processing of dependent information. Therefore, the identification of iris is actually research of dependent evidence fusion. Based on the iris species identification, Figure 1 shows the application process of this method.

4.1.   Application of iris identification based on the proposed method

Step 1. Acquire raw data

The iris data set contains three species, namely, Setosa, Versicolour, and Virginica. We identify flower species through four different attributes: sepal length (SL), sepal width (SW), petal length (PL), and petal width (PW). The data set provides 50 sample data containing these four attributes for each species of iris, with a total of 150 samples. This paper selects iris data set from a machine learning data set ^[38] to demonstrate the application process of the method.

Step 2. Obtain independent principal components through PCA

Eq. (2.7) is used for decentralized processing of the data set, and the matrix $R$ of the four attributes is obtained through Eq. (2.8). The eigenvectors of the matrix $R$ are computed as Table 1.

Calculate the value of the principal component for each iris sample. By substituting the eigenvectors into Eq. (2.11), the expression of the principal components can be derived as Eq. (4.1).

Step 3. Build BBAs of principal components

The data sets of principal components are divided into training sets and test sets to drive the generation of the BBA of each principal component. Different proportions are assigned to the two sets, such as 20% for the training set and 80% for the test set. We assign the proportion of the training set from 20% to 90% for building BBAs, respectively. Furthermore, considering the inherent randomness in BBA generation due to random sampling, we implement a strategy to minimize this randomness. Each simulation in our study involves random reshuffling of the sample order in the original data set 100 times. This approach is designed to eliminate potential biases introduced by the order of data. To ensure a comprehensive evaluation of the model's performance, each experiment is repeated 100 times, allowing us to calculate the average identification accuracy.

This paper generate BBA through the method in Ref. ^[36]. The following is the main process: For a raw data set, each sample can be described by the $l$-dimensional attribute vector $a = \left [a_{b} \right] (b = 1, 2, \dots, l)$. The data of each attribute in the original data set is converted into a normal distribution model. Assume there is a certain sample $S_{i}$ in the training set or test set, take the intersection point between the line $x = x_{b}$ and the normal distribution model, and the longitudinal coordinate of the intersection point is the BBA of the corresponding sample's attribute $a_{b}$ (see Ref. ^[36] for more details).

Step 4. Fuse the BBA and draw a conclusion

Fuse the BBA according to Eq. (2.5), convert the BBA into a probability value through PPT according to Eq. (2.6), and conduct the iris identification experiment. Table 2 shows the average identification accuracy for different training set proportions.

In order to illustrate the role of PCA, we calculate the iris identification accuracy of evidence theory method (i.e., fuse BBAs applying Dempster's combination rule directly without preprocessing), rank correlation coefficient ^[25] and mutual information method ^[26], and compare the proposed method based on PCA with the above methods. Figure 2 represents the average identification accuracy of these four methods for different training set proportions.

From Figure 2, the identification accuracy of evidence theory method is the highest, followed by mutual information method, whereas the proposed method is slightly lower than the previous two, and the accuracy of rank correlation method is the lowest. However, this finding does not indicate that evidence theory method and mutual information method have a better performance in information fusion. The possible reason is that the two methods calculate repeatedly the influence of interrelated parts. This problem is further analyzed and simulated in the next section.

4.2.   Additional experiments

To further illustrate the ability of the method to handle redundant information, redundant attribute data is designed and added to the raw data for simulation.

The attributes to be repeatedly added are selected with reference to the identification reliability of information sources. The identification reliability represents the degree of consistency between the results obtained by direct identification based on the information of a single information source and the actual results. It is defined as follows:

where $F_{SX_{i} }$ is the decision factor, representing the consistency between identified category $\theta _{j}\left \{ {SX_{i}} \right \}$ and real category $\theta _{j}$. Then we obtain the reliability of information source $SX$:

According to Eq. (4.2) and Eq. (4.3), the identification reliability of each attribute of different training set proportions can be obtained as Table 3.

In this simulation, the proportion of training set is set to 60%. From Table 3, the identification reliability of four attributes is as follows: $R_{SL} = 72.81\%, R_{SW} = 56.26\%, R_{PL} = 95.59\%,$ and $R_{PW } = 95.63\%$. It can be seen that the identification reliability of sepal width $R_{SW}$ is the lowest, and that of petal width $R_{PW }$ is the highest.

Case 1. When the attribute $SW$ is redundantly integrated into the original data set, we identify iris species by the four methods. To ascertain the average identification accuracy, the experiment is repeated 100 times following each redundant information integration. Figure 3 shows the changing trend of the identification accuracy of the four methods when the training set accounts for 60% of the raw data and the attribute $SW$ is repeatedly added.

It can be concluded that as the repeated addition of the attribute with the lowest identification reliability, the identification accuracy of evidence theory, rank correlation method, and mutual information method decrease significantly. Among them, evidence theory method dropped to near 0.8, and rank correlation coefficient method and mutual information method dropped to near 0.55. However, the accuracy of the proposed method remains largely stable in the process of repeated attribute addition.

Case 2. When the attribute $PW$ is redundantly integrated into the original data set, we identify iris species by the four methods. To ascertain the average identification accuracy, the experiment is repeated 100 times following each redundant information integration. Figure 4 shows the changing trend of the identification accuracy of the four methods when the training set accounts for 60% of the raw data and the attribute $PW$ is repeatedly added.

Figure 4 shows that as the repeated addition of the attribute with the highest reliability, the identification accuracy of evidence theory method and rank correlation method reached the maximum accuracy value of 0.96 in the second time, and then fluctuated around here. The accuracy of the mutual information method increases rapidly, reaching 0.94 at the first addition and fluctuating also around 0.96 after the second addition. Comparatively, the identification accuracy of the proposed method does not change much in the process of repeated attribute addition.

4.3.   Case studies of other data sets

To enhance the persuasiveness of the proposed method's capability in handling redundant information, we additionally conducts identification experiments on the wine and seeds data sets.

The Wine dataset ^[39] from the University of California, Irvine Machine Learning Repository (UCI Machine Learning Repository) is derived from a chemical analysis of wines grown in Italy, representing three different cultivars. It includes 13 attributes like alcohol content, malic acid, and flavanoids, among others, across 178 instances, categorized into three classes based on the type of cultivar. The Seeds data set ^[40], also from the UCI Repository, is focused on the geometric properties of wheat kernels, comprising 7 attributes such as area, perimeter, and compactness. This data set encompasses 210 instances, divided into three classes representing different wheat varieties: Kama, Rosa, and Canadian.

4.3.1.   Identification accuracy for the wine and seeds data sets

Following the process of the iris identification experiment in Section 4.1, this section calculates the identification accuracy for the wine and seeds data sets using the four methods (see Section 4.1, Step 4), without adding redundant information. The results are shown in Figure 5 and Figure 6, respectively.

Similar to the iris identification experiment, the identification accuracy of the proposed method is not the highest, as this study focuses on the robustness of the method. Experiments involving the addition of redundant information to the wine and seeds data sets are conducted in Section 4.3.2.

4.3.2.   Experiments involving the addition of redundant information

(1) The wine data set

The identification reliability of each attribute in the wine data set is calculated according to Eq. (4.2) and Eq. (4.3), as Table 4. Table 4 indicates that in the wine data set, attribute 7 has the highest reliability in identification, with a score of 0.8150, while attribute 3 shows the lowest reliability, scoring 0.4338. Subsequently, experiments are conducted where attribute 7 and attribute 3 are separately added as redundant information to the original dataset for identification purposes.

Case 1. When the 7th attribute (the highest identification reliability) is redundantly integrated into the original dataset, we identify wine species by the four methods. To ascertain the average identification accuracy, the experiment is repeated 20 times following each redundant information integration. Figure 7 shows the changing trend of the identification accuracy of the four methods when the training set accounts for 60% of the raw data and the 7th attribute is repeatedly added.

Case 2. When the 3rd attribute (the lowest identification reliability) is redundantly integrated into the original data set, we identify wine species by the four methods. To ascertain the average identification accuracy, the experiment is repeated 20 times following each redundant information integration. Figure 8 shows the changing trend of the identification accuracy of the four methods when the training set accounts for 60% of the raw data and the 3rd attribute is repeatedly added.

(2) The seeds data set

The identification reliability of each attribute in the seeds data set is calculated according to Eq. (4.2) and Eq. (4.3), as reported in Table 5. Table 5 indicates that in the seeds dataset, attribute 2 has the highest reliability in identification, with a score of 0.8676, while attribute 6 shows the lowest reliability, scoring 0.5546. Subsequently, experiments are conducted where attribute 2 and attribute 6 are separately added as redundant information to the original data set for identification purposes.

Case 1. When the 2nd attribute (the highest identification reliability) is redundantly integrated into the original data set, we identify seed species by the four methods. To ascertain the average identification accuracy, the experiment is repeated 20 times following each redundant information integration. Figure 9 shows the changing trend of the identification accuracy of the four methods when the training set accounts for 60% of the raw data and the 2nd attribute is repeatedly added.

Case 2. When the 6th attribute (the lowest identification reliability) is redundantly integrated into the original dataset, we identify wine species by the four methods. To ascertain the average identification accuracy, the experiment is repeated 20 times following each redundant information integration. Figure 10 shows the changing trend of the identification accuracy of the four methods when the training set accounts for 60% of the raw data and the 6th attribute is repeatedly added.

From the experiments on adding redundant information to the three data sets, it is observed that with the addition of redundant attributes, the identification accuracies of classical evidence theory, rank correlation coefficient ^[25], and mutual information method ^[26] generally change greatly, and tend to converge toward the inherent reliability of the respective redundant attributes, which is contrary to intuition, while the results of the proposed method are more stable and in accordance with intuition. The other three methods either do not consider the dependence among the information sources (classic evidence theory method) or do not deal with the redundant information among the sources effectively (rank correlation coefficient method and mutual information method), which result in the repeated calculation of the influence of the interrelated parts on the identification result. In contrast, the proposed method has stronger robustness and can effectively deal with redundant information in the process of information fusion.

To further improve the accuracy of the proposed method, the original data can be screened first, and the information sources with particularly low identification reliability can be eliminated, and then the relevant operations of PCA can be carried out.

5.   Conclusion

To solve the problem of dependence in evidence theory, this paper proposes a method of dependent evidence fusion based on PCA. This method starts with the original data of the original evidence source, uses PCA to change the previously related variables into new independent principal component variables, and establishes BBA based on the new variables. Finally, the decision is made by integrating evidence based on Dempster's combination rules. The effectiveness and superiority of the method in this paper are illustrated by the experiments of the iris data sets, the wine data sets, and the seeds data sets.

This method has the following advantages: (1) It is a more reliable and efficient way to analyze the dependence among information sources by using original data, compared with analyzing the dependence by the evidence structure. (2) This paper innovatively proposes a fusion strategy of dependent evidence based on PCA. PCA highlights the main features of data sets and removes redundant information, which can effectively deal with the dependence among information sources and make the fusion results more reasonable and stable.

Our future research endeavors will consider expanding beyond the linear dependencies addressed by PCA, exploring the potential of various nonlinear methods. Alongside this exploration, we intend to assess the compatibility and performance of our method when integrated with established machine learning algorithms, including support vector machine (SVM), k-nearest neighbors (k-NN), extreme gradient boosting (XGB), and random forest. Such integration is anticipated to offer a more comprehensive benchmarking framework, deepening our understanding of our method's adaptability and efficacy across diverse scenarios. In addition, we recognize the importance of a multifaceted evaluation approach. To this end, we aim to refine our evaluation framework by incorporating advanced performance metrics such as area under the receiver operating characteristic curve (AUCROC) and area under the precision-recall curve (AUCPR). These metrics hold significant value, especially in evaluating model performance in situations involving imbalanced data. While it remains to be seen how these modifications will influence our research trajectory, our goal is to ensure a thorough and robust assessment of our method's capabilities, thereby enriching its applicability and contribution to the field.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The work is partially supported by Shanghai Rising-Star Program (Grant No.21QA1403400), Shanghai Natural Science Foundation (Grant No.19ZR1420700), Shanghai Key Laboratory of Power Station Automation Technology (Grant No.13DZ2273800).

Conflict of interest

All authors declare that they have no conflict of interest.