SLeNN-ELM: A shifted Legendre neural network method for fractional delay differential equations based on extreme learning machine

1.   Introduction

In the last decades, it has been noticed that many phenomena usually modeled by ordinary differential equations (ODEs) can be better modeled by delay differential equations (DDEs) ^[1]. Unlike ODEs, the solution of DDEs requires not only information about the current state, but also some information about the previous state. Currently, DDEs play an important role in many practical areas, especially in population dynamics, infectious diseases and chemical kinetics, see ^[2,3,4].

In recent years, fractional differential equations have superior modeling capabilities in various scientific and engineering fields, and fractional delay differential equations (FDDEs) have been widely concerned by researchers ^[5,6]. Numerous studies have shown that FDDEs are applicable to different problems in many fields such as population dynamics, economy, control, medicine, electrodynamics and chemistry ^[7,8].

Since the analytic solution of most FDDEs cannot be represented explicitly, the intensive focus of the researchers is to achieve its numerical solutions ^[9]. Over the years, some numerical techniques for solving FDDEs have been developed. These methods include method of steps ^[10], Adams-Bashforth-Moulton method ^[11], Runge-Kutta-type method ^[12], finite difference method ^[13], Bernoulli wavelets ^[14], Bernstein operational matrix of differentiation method ^[15] and modified Chebyshev wavelet method ^[16]. However, the above techniques do not guarantee that the solutions of differential equations are analytical, and in this respect, neural network methods are superior to classical numerical techniques ^[17].

Differential equations have been solved by neural network methods for more than 20 years ^[18]. Here, we briefly review some critical advances in neural network methods for differential equations. Lagaris et al.^[19] present a method for solving initial and boundary problems for ODEs, coupled ODE systems, and partial differential equations using artificial neural networks as a trial solution. Jafarian et al.^[20] use the generalized sigmoid function as the cost function and the three-layer feedforward structure is considered an iterative scheme for solving linear fractional ODEs. Hou et al.^[21] study the use of neural networks for various pantograph-type functional differential equation problems with a proportional delay term subject to initial or boundary conditions. Shiri et al.^[22] propose a neural network method to solve fractional diffusion equations using an adaptive gradient descent method to minimize the energy function. Recently, Ye et al.^[23] introduce a deep neural network approach to solve time-fractional differential equations in the conformable sense. In addition, other advances in fractional-order time-delay neural networks can be found in ^[24,25,26].

Compared with classical numerical methods, neural network methods offer the following advantages:

● Neural networks trained can obtain the value of the solution of the differential equation at any position in the definition domain.

● Due to the representability of neural networks, the resulting neural network solutions are analytical in essence. That is, these solutions are available in the form of continuously differentiable functions.

● Neural network methods possess high solution accuracy and good generalization properties.

● A similar procedure is employed for different initial and boundary conditions of the differential equations.

● The computational complexity of neural network methods does not increase rapidly with more sampled training points.

Up to present, efforts to invest in FDDEs remain low compared to other fields ^[27]. Furthermore, to the best of our knowledge, no scholars have used neural network methods to solve FDDEs. We therefore aim to propose an accurate and efficient numerical method to solve FDDEs. This is the motivation of this paper. As a family of orthogonal functions, the Legendre family behaves very efficiently in numerical calculations ^[28]. Yang et al. proposed a Legendre neural network for function approximation to avoid the complexity of traditional feedforward neural networks ^[29]. Both Saadatmandi et al.^[30] and Qu et al.^[31,32] studied the methods based on Legendre polynomials for solving fractional differential equations. In addition, Legendre neural network has been successfully applied in channel equalization ^[33] and system identification fields ^[34]. Inspired by these, we propose a shifted Legendre neural network (SLeNN) to solve the following two problems of FDDEs:

$1.$ Constant delay

$2.$ Proportional delay

where $\tau \in (0, 1]$, $\alpha \in (n-1, n]$, $n \in \mathbb{N}$, $x^{(p)}$ represents the $p$-derivative of $x$ and $\lambda_{p} \in \mathbb{R}$ for $p \in \{0, 1, \ldots, n-1\}$, and $\phi(0) = \lambda_0$.

The proposed SLeNN is a functional link neural network (FLNN) based on shifted Legendre polynomials, which enhances the input by shifted Legendre polynomials and skips the hidden layer to improve computing efficiency. The output of SLeNN is used as the trial solution for FDDEs. Unlike expensive gradient-based optimization techniques ^{[18,19,20,31,32]}, we use an improved extreme learning machine ^[35] to solve SLeNN.

The main contributions of this paper are as below:

● For the first time, a single layer functional link neural network based on shifted Legendre polynomials is established for solving the FDDEs, including constant delay and proportional delay.

● The proposed method is a completely non-iterative neural network method with high efficiency.

● Since the proposed method uses shifted Legendre polynomials in place of the hidden layer of the network, it can perform fractional derivatives of FDDEs analytically without any truncation or numerical discretization.

● Different from the usual neural network methods, the proposed method does not need to initialize the parameters of the neural network randomly, ensuring the stability of the results.

The rest of this paper is arranged as follows. Section 2 briefly introduces the properties and related formulas of fractional calculus and shifted Legendre polynomials. We describe the structure of SLeNN in Section 3. In Section 4, specific methods for solving fractional differential equations with constant and proportional delays are presented, respectively. After that, six numerical examples are given in Section 5 to demonstrate the effectiveness and advantages of our method. Section 6 summarizes the main work of this paper.

2.   Mathematical preliminaries

This section first gives some basic definitions and related properties of fractional calculus; see ^[36] for more details.

Definition 2.1. Let $x(t) \in L^1([t_0, T]; \mathbb{R})$. The Riemann-Liouville fractional integral of order $\alpha > 0$ is defined as

where $\Gamma(\cdot)$ denotes the Gamma function.

Definition 2.2. Let $x(t) \in AC([t_0, T]; \mathbb{R})$. The Caputo fractional derivative of order $\alpha > 0$ for a function $x(t)$ is defined as

Since the derivative of a constant is 0, the following property holds in the Caputo sense:

Next, several basic formulas about shifted Legendre polynomials are presented ^[37].

Definition 2.3. Shifted Legendre polynomials $Leg_{T, n}(t)$ of degree $n$ have the following explicit analytical form:

or

Property 2.1. The derivative recurrence relation of shifted Legendre polynomials $Leg_{T, n}(t)$ is given by

where $n \geq 1$.

3.   Shifted Legendre neural network (SLeNN) Model

In this section, we will consider a single layer shifted Legendre neural network model for the present problem. The structure of SLeNN consists of an input neuron, a functional expansion block based on shifted Legendre polynomials and an output neuron. We use shifted Legendre polynomials to expand each input data into several terms. Thus, the shifted Legendre polynomial can be regarded as a new input vector. Let us consider the input neuron $t$ with $m$ training data, which can be denoted as $t = (t_{1}, t_{2}, \ldots, t_{m})^{T}$. The basic formulas of shifted Legendre polynomials are given by Definition 2.3 and Property 2.1. It is worth mentioning that the shifted Legendre polynomials satisfy orthogonality on the interval $[0, T]$.

In this work, we augment the input mode by shifted Legendre polynomials. Here, the input vector $t$ is expanded an $n$-dimensional augmented shifted Legendre polynomial. Then the vector of enhanced mode is obtained by using shifted Legendre polynomials as

The architecture of the SLeNN with the first five shifted Legendre polynomials, input layer (with single node) and output layer (with single node) is shown in Figure 1. We will describe how to solve the FDDE by SLeNN-ELM in the next section.

4.   SLeNN-ELM for solving fractional delay differential equations

In this section, we introduce how to use SLeNN and an improved extreme learning machine to solve fractional differential equations with constant and proportional delays.

At the beginning of this section, for ease of expression, we write the output of SLeNN as the following mathematical expression:

where $w_{i}$ represents the connection weight between the shifted Legendre expansion block and the output layer, $\tilde{x}(t)$ is the output of SLeNN.

According to Eq (2), we can analytically derive the $\alpha$-derivative of $\tilde{x}(t)$:

4.1.   Methodology for solving fractional differential equations with constant delay

We consider the generalized form of fractional differential equations with constant delay, i.e., Eq (1.1). For simplicity, we take $x(t) = \phi(t)$, $t \leq 0$ and $\lambda_{0} = \phi(0)$ as an example. In the proposed method, we do not need to construct the trial solution, but rather separate the adjustable parameters $w_{i}$ in SLeNN. Substituting Eqs (4.1) and (4.2) into Eq (1.1) and using simple mathematical transformations, we can obtain

Now, for different training data of $t$ from the definition domain, we can rewrite the above formula as matrix form $H*w = \beta$ with

where $H_{i}(t) = \frac{\Gamma(i)}{\Gamma(i-\alpha)}Leg_{T, i-1}(t^{\alpha}) - h_{2}(t)Leg_{T, i-1}(t) - h_{3}(t)Leg_{T, i-1}(t-\tau)$ and the information about the vector $\beta$ is already available.

Next, for the initial condition $x(t) = \phi(t), t \leq 0$, we take $m^{\prime}$ training data in $t \leq 0$. Likewise, we can construct the matrix form $H_{IC}*w = \beta_{IC}$, where

When the solution of SLeNN satisfies both Eq (1.1) and initial conditions, the following linear systems can be obtained:

Since we cannot guarantee that $\left(\begin{array}{c}H \\ H_{IC}\end{array}\right)$ is a non-degenerate or square matrix, we choose to find the minimum norm least square solution of the system, namely:

where $A^{\dagger} = \left(A^{T} A\right)^{-1} A^{T}$ is the Moore-Penrose generalized inverse and $w^{*}$ exists and is unique. Finally, we can gain the SLeNN solution by substituting the parameters $w_{i}^{*}$ into Eq (4.1) and regard it as the solution of FDDEs.

4.2.   Methodology for solving fractional differential equations with proportional delay

Here, we consider the fractional differential equations with proportional delay, i.e., Eq (1.2). To make it easy, let's take $x(0) = \lambda_0$ and $x^{\prime}(0) = \lambda_1$ as an example. Similar to Section 4.1, we substitute Eqs (4.1) and (4.2) into Eq (1.2) and move all terms related to $x$ to the left side of the formula,

Now, in order to get the matrix form $G*w = \beta$, we take $m$ number of training points from the definition domain. By exploiting these training points, we have

with $G_{i}(t) = \frac{\Gamma(i)}{\Gamma(i-\alpha)}Leg_{T, i-1}(t^{\alpha}) - g_{2}(t)Leg_{T, i-1}(t) - g_{3}(t)Leg_{T, i-1}(t-\tau)$.

Then we consider two initial conditions $x(0) = \lambda_0$ and $x^{\prime}(0) = \lambda_1$. In order to make the obtained SLeNN-ELM solution satisfy the initial conditions, we also need to construct the matrix form of $G_{IC}*w = \beta_{IC}$ as follows:

After that, the above problem is solved using the Moore-Penrose generalized inverse theory. Similar to Eq (4.4), we gain the parameters of SLeNN by the following formula

Since the target matrix is not square in most cases (depending on the number of training points and the order of the SLeNN), we have to use a pseudo-inverse. The obtained optimal parameters can be used to represent the established SLeNN. Notably, unlike traditional gradient-based neural network training methods, this method does not require any optimization process.

5.   Numerical illustrations

In this section, we use six numerical experiments to verify the accuracy and superiority of the proposed method. All experiments are performed on a computer, which is configured as follows: Intel(R) Core(TM) i5-8265U CPU @ 1.60 GHz 1.80 GHz.

5.1.   Fractional differential equations with constant delay

Example 5.1. Consider the following fractional differential equations with constant delay ^[27,38]:

The exact solution is $t^2$–$t$–$1$. For a more visual comparison, we show the curves of the SLeNN-ELM solution and the exact solution in Figure 2, with the SLeNN-ELM solution in blue and the exact solution in red. Table 1 shows the comparison of the absolute errors of the proposed method and Legendre wavelet method ^[27] at different model parameters. The results show that our method outperforms the Legendre wavelet method. It can be seen that our method is very effective, with errors as low as $O(10^{-16})$.

Example 5.2. We consider the following FDDE ^[38]:

Here, we have $x(t) = t^{2}-t$ as the exact solution. We plot the predicted solution obtained by SLeNN-ELM and compare it with the exact solution in Figure 3. What can be observed is that the SLeNN-ELM method exhibits good accuracy. Table 2 compares the absolute errors of the exact solutions of our method and the method in ^[38]. The root mean squared error and the max absolute error of our method are $1.04 \times 10^{-15}$ and $1.33 \times 10^{-15}$, respectively.

Example 5.3. In this example, we consider the following equation with constant delay ^[6,41,42]:

When $\alpha$ = 3, the exact solution is $e^{-t}$. To see the characteristic of the solution more intuitively, we show in Figure 4 the resulting SLeNN-ELM solution and the exact solution, which are two curves that overlap almost precisely. In Table 3, we list the results of Chebyshev wavelets method ^[41], method in ^[6] and our method with $m = 10$, $n = 10$. Here, in order to match and compare the results reported in the literature above, we only present the absolute error value at $t \in [0, 0.1]$. We observe that the proposed method is more accurate than other methods.

Moreover, to demonstrate the effectiveness of our method more visually, we depict the absolute error by our method for Example 5.3 in Figure 5. From Figure 6, we can find that the solutions obtained by our method vary consistently from fractional to integer order, indicating that the fractional order solutions are accurate.

5.2.   Fractional differential equations with proportional delay

Example 5.4. Consider the following example of fractional differential equations with proportional delay ^[14,27]:

where $t \in [0, 1]$ and $\alpha \in (0, 1]$ with the initial condition $x(0) = 0$.

When $\alpha = 1$, we get $x(t) = te^{-t}$ as the exact solution. We plot the solution obtained by SLeNN-ELM and the exact solution in Figure 7 to facilitate our observation of the characteristics of the solution. To highlight the advantages of the proposed method, we have also compared it with other numerical methods. For $\alpha = 1$, Table 4 shows the absolute errors of our method with $m = 10$, $n = 10$ and other methods: variational iteration method ^[39] with $\hat{m}$ = 6, two-stage order-one Runge-Kutta method ^[40], generalized fractional-order Bernoulli wavelet ^[14] with $k = 2$, $M1 = 6$, $\alpha = 1$ and Legendre wavelet method ^[27] with $k = 2$, $M = 6$.

When $\alpha \in (0, 1)$, the exact solution is unknown. To prove the accuracy of the proposed method, we extend the domain from $t \in [0, 1]$ to $t \in [0, 8]$ and plot the SLeNN-ELM solution at different values of $\alpha$ in Figure 8. The obtained results are in good agreement with those in ^[27].

Example 5.5. As the second example, consider ^[43,44]:

and the exact solution for this example is $t^3$.

In Figure 9, we can visualize the accuracy of the SLeNN-ELM solution by plotting the exact solution and the neural network solution. The comparison of absolute values of errors at grid points selected by our method and method in ^[44] is shown in Table 5. Here, the parameters $m = 8$, $n = 10$ are chose. The error obtained by our method is at least $O(10^{-16})$. Obviously, our method performs better.

Example 5.6. We give a higher-order example of proportional delay. Consider the equation as follows:

with the initial condition $x(0) = x^{\prime}(0) = 0$ and we obtain $x(t) = t^2$ as the exact solution.

The exact solution and the SLeNN-ELM solution are graphed in Figure 10. It can be observed that the two curves are almost perfectly fitted. In Table 6, we show the absolute values of the error using our method with $m = 5$, $n = 10$ and $m = 7$, $n = 10$, respectively. We note that different parameter values both provide a fairly high accuracy for our SLeNN-ELM solutions.

Finally, we list the CPU time required by the proposed and other methods in Table 7. We observe that the proposed method can achieve good accuracy with very high efficiency.

6.   Conclusion

In the present work, we propose a shifted Legendre neural network based on an extreme learning machine algorithm (SLeNN-ELM) to solve fractional differential equations with constant and proportional delays. Neural network methods have many advantages over classical numerical methods in solving differential equations. For example, the solution obtained is analytical in essence. This is helpful for us to better study the properties of the solution of the equation. Different from the traditional neural network method, we use the extreme learning machine algorithm instead of the gradient-based training process, which significantly improves the computational efficiency under the premise of ensuring the accuracy of the solution. In addition, SLeNN is a FLNN, which uses function expansion blocks instead of hidden layers to improve the computational efficiency of the network by reducing the number of parameters. The required calculation time for our method is listed in Table 7. On the other hand, the proposed method can perform fractional derivatives analytically without any truncation or numerical discretization for the fractional problem. More importantly, the proposed method does not need to initialize parameters randomly like other neural network methods, which enables us to obtain a stable solution to any problem. Finally, six examples are given, and the proposed method is compared with other numerical methods. The numerical results show the effectiveness and superiority of our method. In future work, we will consider combining multi-hidden layer neural networks ^[45] and extreme learning machine algorithms to better solve fractional differential equations with different time delays.

Acknowledgments

The authors express their sincere thanks to the anonymous reviewer for his/her careful reading of the paper, giving valuable comments and suggestions. It is their contributions that greatly improve the paper. The authors also thank the editors for their kind help.

This work was supported by the National Natural Science Foundation of China (Nos. 11801452, 11701456), Fundamental Research Project of Natural Science in Shaanxi Province General Project (Youth) (Nos.2019JQ-415, 2019JQ-196), Chinese Universities Scientific Funds(Nos. 2452022104, 2452022105, 2452022372), and Innovation and Entrepreneurship Training Program for College Students of Northwest A & F University (202110712186, 202210712048, S202210712377).

Conflict of interest

The authors declare there is no conflict of interest.