
Citation: Suqing Lin, Zhengyi Lu. Permanence for two-species Lotka-Volterra systems with delays[J]. Mathematical Biosciences and Engineering, 2006, 3(1): 137-144. doi: 10.3934/mbe.2006.3.137
[1] | Zhen-Zhen Tao, Bing Sun . A feedback design for numerical solution to optimal control problems based on Hamilton-Jacobi-Bellman equation. Electronic Research Archive, 2021, 29(5): 3429-3447. doi: 10.3934/era.2021046 |
[2] | Yiming Su, Haiyan Liu, Mi Chen . Robust equilibrium reinsurance and investment strategy for the insurer and reinsurer under weighted mean-variance criterion. Electronic Research Archive, 2023, 31(10): 6384-6411. doi: 10.3934/era.2023323 |
[3] | Julian Gerstenberg, Ralph Neininger, Denis Spiegel . On solutions of the distributional Bellman equation. Electronic Research Archive, 2023, 31(8): 4459-4483. doi: 10.3934/era.2023228 |
[4] | Changling Xu, Huilai Li . Two-grid methods of finite element approximation for parabolic integro-differential optimal control problems. Electronic Research Archive, 2023, 31(8): 4818-4842. doi: 10.3934/era.2023247 |
[5] | Peng Yu, Shuping Tan, Jin Guo, Yong Song . Data-driven optimal controller design for sub-satellite deployment of tethered satellite system. Electronic Research Archive, 2024, 32(1): 505-522. doi: 10.3934/era.2024025 |
[6] | Saeedreza Tofighi, Farshad Merrikh-Bayat, Farhad Bayat . Designing and tuning MIMO feedforward controllers using iterated LMI restriction. Electronic Research Archive, 2022, 30(7): 2465-2486. doi: 10.3934/era.2022126 |
[7] | Sida Lin, Dongyao Yang, Jinlong Yuan, Changzhi Wu, Tao Zhou, An Li, Chuanye Gu, Jun Xie, Kuikui Gao . A new computational method for sparse optimal control of cyber-physical systems with varying delay. Electronic Research Archive, 2024, 32(12): 6553-6577. doi: 10.3934/era.2024306 |
[8] | Suayip Toprakseven, Seza Dinibutun . A weak Galerkin finite element method for parabolic singularly perturbed convection-diffusion equations on layer-adapted meshes. Electronic Research Archive, 2024, 32(8): 5033-5066. doi: 10.3934/era.2024232 |
[9] | Cheng Wang . Convergence analysis of Fourier pseudo-spectral schemes for three-dimensional incompressible Navier-Stokes equations. Electronic Research Archive, 2021, 29(5): 2915-2944. doi: 10.3934/era.2021019 |
[10] | Liangliang Li . Chaotic oscillations of 1D wave equation with or without energy-injections. Electronic Research Archive, 2022, 30(7): 2600-2617. doi: 10.3934/era.2022133 |
It is well-known that the optimal feedback control can be synthesized by the dynamic programming (DP) approach, while finding the optimal control in feedback form is generally argued as the Holy Grail of control theory ([16]). However, before we arrive at the feedback control by this approach, there are two difficulties that need to face. The notion of viscosity solution to partial differential equation (PDE) overcomes the first one that the HJB equation usually has no classical solution regardless of the smoothness of its coefficients ([7,8,9]). Under this framework, such kind of weak solution simultaneously holds the existence and uniqueness. The value function of the optimal control problem is the unique viscosity solution to the HJB equation. Nevertheless, for most of optimal control problems, it is usually impossible to find the analytical viscosity solution, which is the second difficulty to hinder the utilization of the DP approach. The numerical solution is almost the only practically significant way for finding the optimal control via solving the HJB equation. Luckily enough, nowadays there are some literatures available devoted to these topics where the analysis of DP methods for deterministic and stochastic control problems is presented in detail ([3,6,11,12]).
Besides the Bellman DP principle, another milestone work in control theory is contributed to the Pontryagin maximum principle, which establishes a necessary optimality condition satisfied by the investigational optimal control systems consisted of the state equation and its adjoint system ([23,26]). Nevertheless, in essence, the control presented by the Pontryagin maximum principle is of open-loop, which is usually less than satisfactory. Moreover, it is notorious for its sophistication and the fact that it rarely gives an insight into the structure of the optimal controls ([24]). Generally speaking, a feedback control system is considered superior to an open-loop system. Many unnecessary disturbances and noise signals from outside the system can be rejected by using feedback ([4]). Furthermore, even from the numerical solution point of view, for the two-point boundary problem obtained by the necessary condition of optimality, the recognized shooting method has to overcome the difficulty of "guess" for the initial data to start the iterative numerical process ([14,27]), which is generally not an easy job.
People need to make great efforts to develop more efficient algorithms to find the optimal control. Two real reasons for this could not be clearer. One is due to the lack of the analytical solution to the practical optimal control problems. We must numerically solve the investigational problem. Furthermore, the ultimate aim of the control theory is to use it in the real world. It also coincides with the requirement of the information age. The other reason is caused by the curse-of-dimensionality, the inherent defect of the DP approach ([22]). Thanks to many model reduction techniques ([1,3,13,20,19]), this problem can be partially circumvented. It makes some effective algorithms for low dimensional problems, thus initially confined to only "toy'' problems, can be generalized to deal with much more complicated practical optimal control problems. The upwind finite-difference scheme is such a well-adapted algorithm and has been successfully applied to many examples ([13,14,15]). And most of all, its convergence has been rigorously proven in [28].
Certainly it is not easy to find an optimal feedback control by the HJB-based approach and lots of complicated computations are usually involved. Even so, on this topic, there are still many interesting results available in literature, such as [1,5,17,19,22,29], to name just a few. We can unfortunately select only some more relevant research to mention here. For two practical optimal control problems, [14,15] obtain the optimal feedback controls by repeatedly calling the algorithm to the HJB equation. In [2,10,18], the HJB equation is firstly solved to gain the value function. And then the convex combination of numerical value function is applied to obtain the approximation of optimal feedback control. Note that the obtained numerical solution of control function is completely ignored and not directly applied to obtain optimal feedback control. A class of explicit Markov chain approximation methods are introduced and studied in detail in [21]. And these methods are tailored to solving the HJBs for both stochastic and deterministic continuous time nonlinear optimal control problems on finite and infinite horizons. The paper [20] considers a distributed volume control problem in infinite time region and obtains the optimal control by the convex combination of the control values at each point of the polyhedron, which can be interpreted as a special interpolation. The idea of using the interpolation to find the optimal control is also mentioned in [13] but it is kept mostly to one-dimensional linear interpolation for one-dimensional problem. The multi-dimensional problems are not involved in the analysis.
In this paper, we are concerned with a feedback design for numerical solution to the optimal control problem in finite time horizon under the framework of the DPVS approach. To synthesize the optimal feedback control, we solve the HJB equation by the upwind finite-difference scheme. Different from the usual existing algorithms, the numerical control function is interpolated in turn to gain the approximation of optimal feedback control-trajectory pair, which avoid solving the HJB equation repeatedly, thus efficaciously promote the computation efficiency. Moreover, the design can deal with the optimal control problem equipped with multiple controls. And more generally, here, the adopted interpolation is not limited to one-dimensional linear interpolation. Other interpolation techniques including multi-dimensional linear interpolation, cubic spline interpolation, nearest-neighbor interpolation and cubic-Hermite interpolation go well likewise in this new design.
The remainder of this paper is organized as follows. In Section 2, we revisit the procedure of the DPVS approach and present the upwind finite-difference scheme to the HJB equation. An interpolation algorithm then follows, which consists of the core of this paper. The effectiveness of the algorithm is proven in Section 3 by solving five quite distinct examples from each other before some concluding comments are made in Section 4.
Set
{ddty(t)=f(t,y(t),u(t)),t∈(0,T],y(0)=y0, | (2.1) |
where
J(u(⋅))=∫T0L(s,y(s),u(s))ds+h(y(T)), | (2.2) |
where both the running cost
infu∈U[0,T]J(u(⋅)) | (2.3) |
subject to the control system (2.1).
In order to present the DP principle, we consider a family of dynamic optimal control problems. Namely, over
J(u(⋅);t,x)=∫TtL(s,y(s),u(s))ds+h(y(T)), |
subject to
{ddsy(s)=f(s,y(s),u(s)),s∈(t,T],y(t)=x, |
where
v(t,x)=infu(⋅)∈U[t,T]J(u(⋅);t,x),(t,x)∈[0,T)×Rn,v(T,x)=h(x),x∈Rn, | (2.4) |
which is the associated minimal cost functional. It satisfies the DP principle ([3])
v(t,x)=infu(⋅)∈U[t,τ]{∫τtL(s,y(s;t,x),u(s))ds+v(τ,y(τ;t,x))}, |
for any
Then, if the value function
−∂∂tv(t,x)−infu∈U{f(t,x,u)⋅∇xv(t,x)+L(t,x,u)}=0,(t,x)∈[0,T)×Rn,v(T,x)=h(x),x∈Rn, | (2.5) |
where
ˉu(t,x)∈arginfu∈U{f(t,x,u)⋅∇xv(t,x)+L(t,x,u)}, |
we set
u∗(t)=ˉu(t,y∗(t)), | (2.6) |
for almost all
ddty∗(t)=f(t,y∗(t),ˉu(t,y∗(t))),t∈(0,T], | (2.7) |
with
f(t,y∗(t),u∗(t))⋅∇xv(t,y∗(t))+L(t,y∗(t),u∗(t))=infu∈U{f(t,y∗(t),u)⋅∇xv(t,y∗(t))+L(t,y∗(t),u)} |
for almost all
The DPVS approach tells us that once a viscosity solution of the HJB equation (2.5) is obtained numerically, we are able to find the numerical solution of the feedback law by the feedback synthesis above.
Next we revisit the upwind finite-difference scheme to the HJB equation (2.5). Actually, it is such a well-adapted algorithm and has been successfully applied to many examples ([14,13,15]). And most of all, its convergence has been rigorously proven in [28].
Take a polygonal region
vj+1i−vjiΔt+n∑k=1(1+signfjk,i2vjik+−vjiΔxk+1−signfjk,i2vji−vjik−Δxk)fjk,i+Lji=0,uj+1i=arginfu∈U(n∑k=1fk(tj+1,xi,u)vj+1i−vj+1ik−Δxk+L(tj+1,xi,u)), | (2.8) |
for
fjk,i=fk(tj,xi,uji),Lji=L(tj,xi,uji). |
In the following, we construct a HJB-based feedback algorithm for (2.3). Its detailed steps are listed as follows.
Step 1. Initial partition on time and space. Divide the calculation domain into mesh as mentioned in Section 2 above. Denote
Step 2. Compute the approximations of value function and control function. Based on the numerical scheme (2.8), set
Sub-step 2.1. By the same space steps
Sub-step 2.2. Initialize
v0ii=h(xii),u0ii=arginfu∈U(n∑k=1fk(t0,xii,u)v0ii−v0iik−Δxk+L(t0,xii,u)). |
Sub-step 2.3. For
vj+1ii=(1+n∑k=1γk|fjk,ii|)vjii−n∑k=11+signfjk,ii2γkfjk,iivjiik++n∑k=11−signfjk,ii2γkfjk,iivjiik−−ΔtLjii,uj+1ii=arginfu∈U(n∑k=1fk(tj+1,xii,u)vj+1ii−vj+1iik−Δxk+L(tj+1,xii,u)). |
Sub-step 2.4. Choose the numerical solution of value function and control function over
Step 3. Compute the optimal pairs
Sub-step 3.1. Calculate the
uj=˜u(tj,yj). |
Sub-step 3.2. Calculate the
yj+1=yj−f(tj,yj,uj)Δt. |
With the known
Sub-step 3.3. Iterate for the next time instant. Set
Remark 1. In Sub-step 3.1, we can also adopt other interpolation methods, such as polynomial interpolation, average interpolation, the Newton interpolation, Lagrange's interpolation and so on ([25]). Similarly, other difference method can be used to discrete state equation in Sub-step 3.2.
This section is devoted to five numerical examples to verify the effectiveness of the algorithm. The simulations are implemented by using MATLAB programme in a desktop computer with a core process (i5-6600M) and 8GB of random access memory.
Example 1. Consider the following control problem ([31])
{˙y(t)=y(t)u(t), t∈(0,1],y(0)=y0,J(u(⋅))=−y(1),minu(⋅)∈UJ(u(⋅)), |
where
−∂∂tv(t,x)−infu∈[0,1]{xu⋅∇xv(t,x)}=0,(t,x)∈[0,1)×R,v(1,x)=−x,x∈R. |
Its exact value function
v(t,x)={−xe1−t, if x>0,−x, if x≤0, |
and the control function is
ˉu(t,x)={1, if x>0,0, if x≤0. | (3.1) |
Here, we choose the region
Moreover, in order to show the performance of the presented interpolation algorithm, two existing algorithms are additionally chosen to solve Example 1, simultaneously. The corresponding results are also given in Figures 1 and 2. And the comparisons on numerical simulations are listed in Table 1.
Algorithm | Initial state | Control | Trajectory | CPU(s) |
Interpolation | 6.6000e-05 | 3.3001e-05 | 0.005615 | |
6.6000e-05 | 0.016695 | 0.005484 | ||
Algorithm 1 | 6.6000e-05 | 3.3001e-05 | 0.015370 | |
6.6000e-05 | 0.016695 | 0.016507 | ||
Algorithm 2 | 6.6000e-05 | 3.3001e-05 | 0.025182 | |
6.6000e-05 | 0.016695 | 0.025733 |
Actually, one chosen algorithm is the minimisation of the Hamiltonian. People can refer to Algorithm 3.2 in [13] for details. For convenience, we use Algorithm 1 to denote it. In the other one algorithm, the optimal feedback control is also gained by minimizing the Hamiltonian function. But it is different from Algorithm 1, in which the value function is obtained by the convex combination of numerical value function ([18]). We name it as Algorithm 2.
When
For these three algorithms, Table 1 lists the computed errors in the maximum norm between the numerical solutions and exact solutions including the optimal feedback control
Moreover, the CPU times are listed in Table 1. They are the computing time of these three algorithms after we obtain the numerical control function and numerical value function by upwind finite-difference method. By comparing the CPU times, we find that the interpolation algorithm takes the least time than Algorithms 1 and 2 with the similar error estimation. It illustrates that linear interpolation is meaningful for finding an approximation of the optimal feedback control, although the control function (3.1) is discontinuous. In all, numerical results shows that the interpolation algorithm is not only feasible but also efficient for gaining the optimal feedback control.
Example 2. Consider the following control problem ([30])
{˙y(t)=y(t)+u(t), t∈(0,1],y(0)=1,J(u(⋅))=12∫10u2(t)dt+12y2(1),minu(⋅)∈UJ(u(⋅)), |
where
−∂∂tv(t,x)−infu∈R{(x+u)⋅∇xv(t,x)+12u2}=0,(t,x)∈[0,1)×R,v(1,x)=12x2,x∈R. | (3.2) |
The exact solution to the HJB equation (3.2) is
v(t,x)=x21+e2(t−1), |
and the control function is
ˉu(t,x)=−2x1+e2(t−1). | (3.3) |
With (2.6), (2.7) and (3.3), we obtain the optimal feedback control
Now we numerically solve Example 2. Choose the computational region
Error | Control | Trajectory |
0.025896 | 0.017297 |
In addition, in Examples 1 and 2, the control problems are all one-dimensional problems. Next, we present other two optimal control examples with two space dimensions.
Example 3. Consider the following two-dimensional control problem ([29]) with two state variables and one control.
{(˙x(t)˙y(t))=(x(t)y(t))u(t), t∈(0,1],x(0)=x0, y(0)=y0,J(u(⋅))=−x(1)−y(1),minu(⋅)∈UJ(u(⋅)), |
where
−∂∂tv(t,x,y)−infu∈[0,1]{xu⋅vx(t,x,y)+yu⋅vy(t,x,y)}=0,(t,x,y)∈[0,1)×R2,v(1,x,y)=−(x+y),(x,y)∈R2. | (3.4) |
In this example, the exact solution of the value function is
v(t,x,y)={−(x+y)e1−t,if x+y>0,−(x+y),if x+y≤0, |
and the control function is
ˉu(t,x,y)={1, if x+y>0,0, if x+y≤0. | (3.5) |
From (2.6), (2.7) and (3.5), when initial states
Next we use the interpolation algorithm to solve the HJB equation (3.4) in the range
From Figures 4 and 5, we see that, even with different initial values, the images of numerical solutions agree well with the exact solutions for the optimal feedback control and trajectory.
Example 3 shows that, even for two-dimensional control problem, the linear interpolation is meaningful for finding an approximation of the optimal feedback control. On the other hand, we find that the discontinuity of the control function (3.5) does not hinder us to find the optimal feedback control.
Furthermore, all three examples above are equipped with one control. And only the linear interpolation is adopted in the algorithm. And to show the effectiveness and efficiency of the interpolation algorithm in an even better fashion, we consider an optimal control problem with two states and two controls in Example 4. And four interpolations including linear interpolation, cubic spline interpolation, nearest-neighbor interpolation and cubic-Hermite interpolation are successively adopted in the investigation.
Example 4. Consider the following optimal control problem with two state variables and two controls.
{(˙x1(t)˙x2(t))=(x1(t)+u1(t)u2(t)), t∈(0,1],x1(0)=x10, x2(0)=x20,J(u1(⋅),u2(⋅))=12∫10[u21(t)+u22(t)+x22(t)]dx+12x21(1),min(u1(⋅),u2(⋅))∈UJ(u1(⋅),u2(⋅)), | (3.6) |
where
−∂∂tv(t,x1,x2)−inf|u1|,|u2|,|u1+u22|≤1{(x1+u1)⋅vx1(t,x1,x2)+u2⋅vx2(t,x1,x2)+12u21+12u22+12x22}=0,(t,x1,x2)∈[0,1)×R2,v(1,x1,x2)=12x21,(x1,x2)∈R2. | (3.7) |
Here we choose the computational region
Moreover, in this example, the exact value function
Errors | Control | Trajectory |
6.6000e-05 | 4.6671e-05 | |
6.6000e-05 | 0.023611 |
Then, in order to investigate the influence of different interpolation on the interpolation algorithm, we provide the error results of four interpolation cases for this example. In Table 4, we list the computational errors in the maximum norms between the numerical solutions and exact solutions including optimal control
Error | Control | Trajectory |
Linear | 0.038706 | 0.02681 |
Cubic spline | 0.038706 | 0.026801 |
Nearest-neighbor | 0.069388 | 0.030588 |
Cubic-Hermite | 0.038706 | 0.026804 |
In the linear interpolation case, they are
Let me emphasize that, for four examples above, all optimal controls are continuous, while the case of optimal control with jumps is not considered. Next, we use the interpolation algorithm to solve a control problem in which optimal control has jumps. We attempt to show that the interpolation algorithm is also feasible for this kind of problems.
Example 5. Consider the following optimal control problem with two state variables and one control.
{(˙x1(t)˙x2(t))=(x2(t)−x1(t)+14u(t)), t∈(0,2π],x1(0)=0, x2(0)=0,J(u(⋅))=x2(2π),min(u(⋅))∈UJ(u(⋅)), | (3.8) |
where
−∂∂tv(t,x1,x2)−inf|u|≤1{x2⋅vx1(t,x1,x2)+(−x1+14u)⋅vx2(t,x1,x2)}=0,(t,x1,x2)∈[0,2π)×R2,v(2π,x1,x2)=x2,x2∈R. |
Here we choose the computational region
u∗(t)={−1,0≤t<π2,1,π2≤t<3π2,−1,3π2≤t≤2π, | (3.9) |
and
(x∗1(t)x∗2(t))={14(cost−1−sint),0≤t<π2,14(cost−2sint+1−2cost−sint),π2≤t<3π2,14(cost−4sint−1−4cost−sint),3π2≤t≤2π. |
From (3.9), we see that the optimal control is not continuous. It has two jumps in
In this paper, we construct an interpolation algorithm for finding an approximation of optimal feedback control. And five experimental examples with different constraints (even nonlinear constraints) on control are successfully solved to demonstrate the effectiveness of the presented interpolation algorithm. The numerical results show that the algorithm with interpolation is very efficient for solving practical optimal control problems.
The authors would like to thank the editor and the anonymous referees for their very careful reading and constructive suggestions that improve the manuscript substantially.
1. | Sida Lin, Lixia Meng, Jinlong Yuan, Changzhi Wu, An Li, Chongyang Liu, Jun Xie, Sequential adaptive switching time optimization technique for maximum hands-off control problems, 2024, 32, 2688-1594, 2229, 10.3934/era.2024101 | |
2. | Vadim Azhmyakov, Warodom Werapun, Jakapan Suaboot, On the Hamilton–Jacobi-Bellman approach to the relaxed optimal feedback control problems, 2025, 00160032, 107667, 10.1016/j.jfranklin.2025.107667 |
Algorithm | Initial state | Control | Trajectory | CPU(s) |
Interpolation | 6.6000e-05 | 3.3001e-05 | 0.005615 | |
6.6000e-05 | 0.016695 | 0.005484 | ||
Algorithm 1 | 6.6000e-05 | 3.3001e-05 | 0.015370 | |
6.6000e-05 | 0.016695 | 0.016507 | ||
Algorithm 2 | 6.6000e-05 | 3.3001e-05 | 0.025182 | |
6.6000e-05 | 0.016695 | 0.025733 |
Error | Control | Trajectory |
0.025896 | 0.017297 |
Errors | Control | Trajectory |
6.6000e-05 | 4.6671e-05 | |
6.6000e-05 | 0.023611 |
Error | Control | Trajectory |
Linear | 0.038706 | 0.02681 |
Cubic spline | 0.038706 | 0.026801 |
Nearest-neighbor | 0.069388 | 0.030588 |
Cubic-Hermite | 0.038706 | 0.026804 |
Algorithm | Initial state | Control | Trajectory | CPU(s) |
Interpolation | 6.6000e-05 | 3.3001e-05 | 0.005615 | |
6.6000e-05 | 0.016695 | 0.005484 | ||
Algorithm 1 | 6.6000e-05 | 3.3001e-05 | 0.015370 | |
6.6000e-05 | 0.016695 | 0.016507 | ||
Algorithm 2 | 6.6000e-05 | 3.3001e-05 | 0.025182 | |
6.6000e-05 | 0.016695 | 0.025733 |
Error | Control | Trajectory |
0.025896 | 0.017297 |
Errors | Control | Trajectory |
6.6000e-05 | 4.6671e-05 | |
6.6000e-05 | 0.023611 |
Error | Control | Trajectory |
Linear | 0.038706 | 0.02681 |
Cubic spline | 0.038706 | 0.026801 |
Nearest-neighbor | 0.069388 | 0.030588 |
Cubic-Hermite | 0.038706 | 0.026804 |