Sufficiency for singular trajectories in the calculus of variations

1.   Introduction

The Sigma-Pi-Sigma neural network (SPSNN) is a feed-forward neural network composed of Sigma-Pi units, which can be used to achieve a static mapping of multi-layer neural networks ^[1,2,3]. In ^[4], a new model which can be regarded as a subclass of networks on Sigma-Pi units was considered, and the authors showed the origin of the Kronecker product representation from the classical Sigma-Pi units. In ^[5], a Sigma-Pi network and a new arithmetic were proposed, by which the output representation self-organizes to form a topographic map, whose main contribution was to solve the frame of reference transformation issues through unsupervised learning. Furthermore, in ^[6,7,8], the approximation, convergence performance, and generalization ability of sparse Sigma-Pi network functions were studied, and it was shown that the new algorithms were more efficient than those in the existing literatures.

It is widely recognized that neural network optimization has emerged as a highly significant research topic in recent times. Study on neural network optimization consists of two main topics: One is weight optimization, that is, for a given network structure, select the appropriate learning method to seek the optimal weight such that the training error and the generalization error are small enough ^[9,10]. The second is structural optimization, namely the selection of appropriate activation function, network layer number, connection mode, and so on ^[11,12,13]. But, the research on neural network structure optimization is far less rich than that on weight optimization. On the other hand, there is no literature showing that more hidden layer neurons yield better generalization ability.

When it comes to neural networks, the number of neurons is a crucial factor. There are two common methods for determining the size of networks. The first method is the growing method, where the network starts with a smaller size and new hidden neurons are added during the training process ^[14]. Another method is the pruning method ^{[15,16,17,18,19]}, which begins with a larger network and eventually removes redundant nodes.

These kind of algorithms separates weight learning and weight training, which is inefficient. There are also many slightly more complex algorithms, which further introduce various mechanisms like particle swarm optimization ^[20], genetic algorithms ^[21], eigenvalue analysis ^[22], statistical analysis, and synthetic minority over-sampling techniques ^[23,24], so as to enhance the sparsification efficiency. The disadvantage of these algorithms is that the program is complex and the calculation is large.

An appropriately sized network structure is instrumental in enhancing efficiency. Overfitting poses a significant challenge during network training and is particularly problematic for deep neural network learning ^[25]. Consequently, researchers have extensively explored various forms of sparse regularization techniques, highlighting their indispensability.

Recent years, $L_{p}$ regularization is diffusely used to solve variable selection and parameter estimation problems in machine learning. This regularization method takes the form

where $\widehat{E}$ is the normal error function, $\|W\|_{p} = (\sum_{i}|W_{i}|^{p})^{1/p}$ denotes the p-norm, $\tau$ is the penalty coefficient, and $\|\cdot\|$ represents the euclidean norm. This regularization term is also named the penalty term.

In general, there are several common forms of regularization: weight elimination, weight decay, and approximate smoothing ^{[26,27,28,29,30,31,32]}. Among them, weight elimination is widely used as the penalty term in pruning feedforward neural networks, mainly to reduce unnecessary connections or optimize the network weights ^[33]. A little more detailed introduction for different penalty terms are as follows.

For different values of $p$, the $L_{2}$ norm to the standard error function makes it more optimized ^{[34,35,36,37]}. This practice is called $L_{2}$ regularization, and the form is shown as follows:

where $\|W\|^2_{2}$ denotes the penalty term, and the $L_{2}$ norm solution is popular because of its special relationship with the normal distributions. It can serve as brute force to avoid excessive weights. But, unfortunately, it is not sparse. This means that, during the training process, the $L_{2}$ regularization can not drive unnecessary weights to zero.

The $L_{0}$ regularization term is the ordinary method for feature extraction and variable selection. Constrained by the number of coefficients, the $L_{0}$ regularizer produces the most sparse solutions, which are difficult to calculate, and it is a combinatory optimization problem ^[38,39].

As an alternative to the $L_{0}$ regularization term, the increasingly important $L_{1}$ regularization term (Lasso) has become popular since it just needs to resolve a quadratic programming problem ^[40]. In ^[41], the $L_1$-norm was combined with the capped $L_1$-norm to indicate the amount of information collected by the filter and the control regularization. The $L_1$ regularization penalty function is generally denoted as follows:

It was shown in ^[42,43] that, although these algorithms can generate an alternate neural network structure, they do not offer a unified neural network framework to solve a class of problems.

In order to explore more appropriate neural network structures to get over the obstacles posed by suboptimal models, we propose a new SPSNN algorithm based on the $L_1$ penalty and the $L_2$ penalty to deal with complex and varied tasks within a unified framework to improve the robustness and generality of the model. Based on the $L_1$ norm, an $L_2$ norm is presented in SPSNNs to promote the population sparsity effect to select the relevant hidden node population. Therefore, our proposed variant algorithm benefits from ridge regression and the tendency towards sparse solutions of the $L_1$ penalty, which generates a more suitable neural network structure than using one of the regular terms alone, and the elastic net algorithm can also be used to solve for these hybrid penalties ^[44].

In this paper, the penalty method is considered in the case of selecting the weights, which not only overcomes the shortcomings of the $L_1$ and $L_2$ norms, (both $L_1$ and $L_2$ penalties are used in the same minimization problem), but also has good generalization ability and sparsity. Thus, the mathematical expression for this hybrid penalty as an error function can be expressed as follows:

where the tuning constants $\tau_{1}$ and $\tau_{2}$ are fixed and non-negative. The role of $\tau_{1}$ provides a choice of variables through a sparse vector, and $\tau_{2}$ ensures a unique solution and leads to a grouping effect. A penalty term is added that is a convex combination of the $L_1$ norm $\|W\|_{1}$ and the $L_2$ norm $\|W\|_{2}^{2}$ of the parameter $W$.

During the training progress, the usual mixed regularization terms are not differentiable at the origin, which usually give rise to oscillation phenomenon. Therefore, we propose a new smoothing algorithm to get over these difficulties. That is, we can use the smoothing technique of the weight in the neighborhood of the origin of the error function instead of the absolute value of the weight. The main contribution and novelty of this article are as follows:

(1) In this article, in order to obtain the optimal architecture with good generalization performance, we will eliminate the weights in the hidden layer by using the idea of elastic net regularization. It means that, by incorporating the $L_1$ and $L_2$ regularization terms, this novel algorithm not only eliminates unnecessary weights but also performs pruning of the network structure in the hidden layers. This pruning reduces the size of the network, leading to an effective optimization of the network structure.

(2) We propose an SPSNN based on elastic net regularized batch gradient methods to obtain an optimal architecture with good generalization performance. By means of smoothing technology, we effectively get over the oscillation phenomenon in the process of network learning.

(3) To test the accuracy and robustness, we apply the proposed method to a large number of regression and classification tasks and compare it with other algorithms that have good sparsity and generalization capabilities.

The rest of the article is arranged as follows: In Section 2 the SPSNN is presented, the batch gradient algorithm for this model is depicted in Section 3, and the novel pruning algorithm based on $L_1$ plus $L_2$ penalty term is depicted for more details. Some numerical experiments are provided in Section 4, and finally some conclusions are made.

2.   Sigma-Pi-Sigma neural networks

Next, we mainly depict the fabric of SPSNNs, which are composed of an input layer, a hidden layer of summing nodes($\sum_{1}$), a hidden layer of product nodes($\prod$), and output layer($\sum_{2}$). P, N, Q, and 1 represent the number of the nodes of the input layer, $\sum_{1}$ layer, $\prod$ layer, and $\sum_{2}$ layer, respectively (see Fig. 1)

We can use $W_{0} = (w_{01}, w_{02}, \cdots w_{0Q})^T\in{\mathbb{R}^Q}$ to express the weight vector which connects $\sum_{2}$ layer and the $\prod$ layer. Let the weight vector connecting the $\prod$ layer and the $\sum_{1}$ layer be fixed as 1. Meanwhile, using $W_{n} = (w_{n1}, w_{n2}, \cdots w_{nP})^T$ as the weight vector which connects the input layer and the $nth$ node of the $\sum_{1}$ layer, we can set

Let us use $X = (x_1, x_2, \cdots x_P)\in \mathbb{R}^P$ to describe the input of the networks. Assume that $g:\mathbb{R}\longrightarrow \mathbb{R}$ is a given sigmoid activation function for the $\sum_{1}$ layer. We denote the output vector $\xi\in{\mathbb{R}^N}$ of the $\sum_{1}$ layer as

where $\cdot$ denotes the inner product between vectors.

Similarly, we take $\delta = (\delta_1, \delta_2, \cdots \delta_Q)^T$ to indicate the output vector of the $\prod$ layer. The nodes of the network are connected in two different ways. One is fully connected between the $\sum_{1}$ layer and the $\prod$ layer, and the other one is sparsely connected. The difference between them lies in the number of product nodes: the former has $2^N$ product nodes, and the latter is less than $2^N$. We can clearly see the structure of the SPSNNs in Figure 1.

Here, $\mathcal{H}_q(1\leq q\leq Q)$ represents the set of nodes in the $\sum_{1}$ layer connected with the q-th product node, while the set of all product nodes connected with the n-th node in the $\sum_{1}$ layer is represented by $\mathcal{F}_n(1\leq n\leq N)$. We can suppose an arbitrary muster italicize, and $\varphi(a)$ is the number of elements in muster italicize, and we have

which will be used later.

We can compute the output vector $\delta\in \mathbb{R}^Q$ in the $\prod$ layer by

To make convention, we denote $\prod_{i\in\mathcal{H} q}\xi_i = 1$ when $\mathcal{H}_q = \varnothing$. In the SPSNNs, the final output can be written as

3.   Batch gradient algorithm for Sigma-Pi-Sigma neural networks with elastic net regularization

3.1.   Batch gradient algorithm for Sigma-Pi-Sigma neural networks

We introduce the batch gradient algorithm for SPSNNs. Let $\{X^l, O^l\}_{l = 1}^L\subset \mathbb{R}^P\times \mathbb{R}$ be the given set of the training samples, where $X^l$ denotes the $lth$ input sample and the $O^l$ is the $lth$ corresponding ideal output. Let $y^l\in R$ be the real output for each input $X^l$. The conventional square error function can be given as

where $g_l(z) = \frac{1}{2}(g(z)-O^l)^2, z\in\mathbb{R}, 1\leq l\leq L.$

For convenience, we need to provide the following forms:

and thus, by virtue of

and some calculation, we can gain

It follows from $\delta_{q}^{l} = \prod_{i\in\mathcal{H}_Q}g(W_{i}\cdot X^{l})$ that

Then, from the above equality and (3.2), we get

3.2.   Batch gradient algorithm for Sigma-Pi-Sigma neural networks with elastic net regularization

To simplify the network structure, we add the elastic net regularization to optimize the network on the group level. So, we can get the corresponding form of the error function

where $\tau$ and $\alpha$ are the tuning parameters that control the performance of the penalty term. With the gradual increase of $\alpha$, the $L_1$ regular term dominates, and the error function gets closer to Lasso regression; with the gradual decrease of $\alpha$, the $L_2$ regular term dominates, and the error function gets closer to ridge regression. In particular, the error function is equivalent to that of ridge regression at $\alpha = 0$, and the cost function is equivalent to that of Lasso regression at $\alpha = 1$. Let $\tau_1 = \tau\cdot\alpha$ and $\tau_2 = 2\tau\cdot(1-\alpha)$. Then, we have

The gradient of the error function with respect to $W_0$ and $W_n$ are, respectively, given as

and

We notice that elastic net regularization in (3.4) is combined with $L_1$ norm regularization and $L_2$ norm regularization. It is clear that (3.4) is not differentiable at the origin, which will yield the oscillation phenomenon, and we propose a smoothing approximation method to overcome this problem caused by the non-smoothness. For any limited dimensional vector $\mathbf{u}$ and a fixed constant $\gamma > 0$, we can define a smoothing function of $\|\mathbf{u}\|$ as follows:

We use (3.7) to approximate the elastic net regularization in (3.4). Furthermore, the gradient of $h(\mathbf{u}, \gamma)$ with respect to the vector $\mathbf{u}$ is given as follows:

Accordingly, the error function (3.4) can be rewritten as

According to (3.8), we can get a smoothing elastic net Sigma-Pi-Sigma neural network as

where $l = 1, 2, \cdots, L$.

Beginning with an arbitrary initial weight vector $W^0$, by the following iterative formula we define the weight sequence

where $\eta$ represents the learning rate.

4.   Simulation results

In this section, the performance of the models with no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer $(OL_{1/2})$, the smoothing $L_{1/2}$ regularizer $(SL_{1/2})$, and the original group lasso regularizer $(OGL)$ algorithms are compared with the smoothing elastic net regularizer algorithm $(SGL)$ by using four examples: classification problem, parity problem, function approximation problem, and prediction problem.

4.1.   Classification problem

In this example, we choose 8 benchmark data sets from the UCI machine learning repository to test the performance of the new algorithm $(SGL)$, and compare it with the no regularizer, the $L_{2}$ regularizer, the $OL_{1/2}$, the $SL_{1/2}$, and the $OGL$ algorithms.

Table 1 presents the main characteristics of the relevant data sets, which includes the size of datasets, attributes, categories, and sizes of the training and testing sets, where the dataset is randomly partitioned into two subsets: 70% for training and 30% for testing.

As described at the beginning of this paper, we learn the structure of SPSNNs (see Figure 1). Then, we select ${P = 13}$, ${N = 4}$, ${Q = 16}$, and 1, representing the number of the nodes of the input, the ${\sum_{1}}$, the ${\prod}$, and the ${\sum_{2}}$ layers, respectively. For each learning algorithm, the initial weights are randomly selected in the interval ${[-0.5, 0.5]}$, the learning rate ${\eta}$ is 0.0028, and the regular factor ${\tau}$ is 0.001, and we conduct 20 trials for every data set to compare the performance of different algorithms.

To assess the performance of the smoothing elastic net regularizer, based on each data set, we compare the number of remaining hidden neurons after pruning $(RNN)$, the training accuracy testing accuracy, and training time for each algorithm, and all experimental results are recorded in Table 2. From the table, it can be observed that the training accuracy has slightly improved, while the testing accuracy has increased by approximately 1% to 3%. We can find our proposed smoothing elastic net regularizer is superior to the no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer, the smoothing $L_{1/2}$ regularizer, and the original group Lasso regularizer algorithms.

In addition, we have also compared our approach with other existing methods. In ^[45], the authors considered the group Lasso regularization method on the Sigma-Pi-Sigma neural network. In ^[46], the authors applied the group $L_{1/2}$ regularization term on high-order neural networks. Our proposed elastic net regularization method is on par with these approaches.

4.2.   5-bit parity problem

For the parity problem, there is an input set of ${2^n}$ samples in n-dimensional space, and every sample is an n-bit binary vector. We consider a 5-bit parity problem which has an input set with ${2^5}$ samples in 5-dimensional space; the ideal output equals to 1 if the number of 1 in the samples is odd, otherwise it equals to zero. Here, using the above method, we test the performance of our proposed smoothing elastic net regularizer.

Similarly, we can study the structure of SPSNNs. We select ${P = 13}$, ${N = 4}$, ${Q = 16}$, and 1 for the number of the nodes of the input, ${\sum_{1}}$, ${\prod}$, and ${\sum_{2}}$ layers, separately. In the interval ${[-0.5, 0.5]}$, the initial weights are randomly selected, the learning rate ${\eta}$ is 0.0045, and the regular factor ${\tau}$ is 0.001. For each learning algorithm we carry out 20 experiments, and we train up to 40, 000 steps or we stop once the error is less than ${1e-4}$. So, as to assess the sparsity and convergence of the smoothing elastic net regularizer, we compare the average error $(AVE)$ and the number of remaining hidden neurons after pruning $(RNN)$ with the no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer, the smoothing $L_{1/2}$ regularizer, the original group Lasso regularizer, and the smoothing elastic net regularizer, which are listed in Table 3.

The results show that the proposed smooth elastic net regularizer outperforms the no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer, the smoothing $L_{1/2}$ regularizer, and the original group Lasso regularizer.

Figure 2(a) shows the error performance of the original group Lasso regularizer and smoothing elastic net regularizer via the 5-bit parity problem. Figure 2(b) shows that the norm of the gradient curve of the error function, based on the 5-bit parity problem, approaches a small positive constant. This indicates that the smoothing elastic net regularizer removes the oscillation of occurring in the original group Lasso regularizer in the learning process.

4.3.   Function approximation problem

In this example, we study the multi-dimensional Gabor function to compare the approximation performance of the above algorithms.

As described at the beginning of this paper, we learn the structure of the SPSNNs. Then, we select ${P = 13}$, ${N = 4}$, ${Q = 16}$, and 1 for the number of the nodes of the input layer, ${\sum_{1}}$, ${\prod}$, and ${\sum_{2}}$ layers, separately.

In this experiment, we select 169 training samples from an evenly spaced ${6\times6}$ grid on the square ${-0.5\leq x \leq 0.5}$ and ${-0.5\leq y \leq 0.5}$. In the interval ${[-0.5, 0.5]}$, the initial weights are randomly selected, the learning rate ${\eta}$ is 0.0028, and the regular factor ${\tau}$ is 0.001. For each learning algorithm we carry out 20 experiments, and we train up to 40, 000 times or until the error is less than ${1e-4}$ stop iterations.

To assess the sparsity and convergence of the smoothing elastic net regularizer, we compare the average error $(AVE)$ and the number of remaining hidden neurons after pruning $(RNN)$ with the no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer, the smoothing $L_{1/2}$ regularizer, the original group Lasso regularizer, and the smoothing elastic net regularizer, which are shown in Table 4. We can see our proposed smoothing elastic net regularizer is superior to the no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer, the smoothing $L_{1/2}$ regularizer, and the original group Lasso regularizer.

For each learning algorithm, we show the error function and the norm of gradient of one of the 20 experiments after 40, 000 epochs in Figures 3–5. Figure 3(a) shows the oscillation phenomenon of no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer and the original group lasso regularizer. Figure 3(b) shows the error curve of the smoothing $L_{1/2}$ regularizer and the smoothing elastic net regularizer. Figure 4(a) shows the norm of gradient curve of the no regularizer, the $L_{2}$ regularizer, the original $L_{1/2}$ regularizer, and the original group Lasso regularizer. Figure 4(b) shows the norm of the gradient curve of the smoothing $L_{1/2}$ regularizer, and the smoothing elastic net regularizer. Obviously, it approaches to a small positive constant. Figure 5 shows a typical performance for one of 20 experiments, and we can see that it has a good approximation effect compared with other algorithms. In each learning algorithm for the same parameters, we get the corresponding results. We can see the learning method with the smoothing elastic net regularizer converges faster than other learning methods, and the smoothing elastic net regularizer method overcomes the numerical oscillation phenomenon. It also shows that during the iterative process the error function curves are monotonically decreasing and converge to zero, which also validates our theoretical results.

4.4.   Prediction problem

To verify the effectiveness of the algorithm further, this part takes an interval shield tunneling project of Metro Line 9 in Zhengzhou City, China, as an example. In order to monitor the impact of the subway shield tunneling process on the surface buildings and structures, 10-meter intervals are used to set up settlement observation points in advance in each shield tunneling section, and the surrounding structures and buildings are monitored. JGC1, the closest settlement observation point of the signal tower to the right line of the shield structure, is selected as the objective of study, and the Leica DNA03 level is used to collect data 10 days before the shield structure is excavated to the closest point of JGC1, for a total of 40 days, and the frequency of observation is once a day. In this experiment, we use the data of the first 30 days as the training data set and the data of the last 10 days as the test data set (see Table 5).

In this experiment, we learn the structure of the SPSNNs. Then, we select $P = 5$, $N = 4$, $Q = 16$, and $1$ for the number of the nodes of the input layer, $\sum_1$, $\sum_2$, and the output layer, respectively. We use the sigmod activation function at the $\sum_1$ and output layers, respectively, and our stopping criteria in this experiment is an error of less than $1\times10^{-5}$ or $5000$ iterations.

Figure 6 is the error curve of the training set with $5000$ iterations, in which red is the error curve without the regularization term and blue is the error curve with the smoothing elastic net regularization, it can be obtained that the error of the network with the smoothing elastic net regularization decreases faster, and after $500$ iterations, the error is smaller than that without the regularization term, which precisely verifies the theoretical results of this paper and the effectiveness of the proposed algorithm.

5.   Conclusions

In this paper, a new batch gradient algorithm for SPSNNs with an $L_1$ plus $L_2$ regularization algorithm is proposed as an effective weight pruning technique. It can handle multi-output regression and multi-class classification problems within a unified framework. This algorithm obtains good performance in both Lasso and ridge regression, penalizing the weights by reducing the weight vectors to zero, which is more efficient than other various pruning strategies. Moreover, the theoretical results and the advantages of this algorithm are also illustrated by numerical experiments.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Author contribution

Conceptualization, methodology, original draft preparation, J. Jiao; software, editing, K. Su.

Conflict of interest

All authors declare there is no conflict of interest.