Stochastic linear quadratic optimal tracking control for discrete-time systems with delays based on Q-learning algorithm

1.   Introduction

Let $\Omega = \{z\in \mathbb R^n: R_1 < |z| < R_2, R_1, R_2 > 0\}$. In this work we study the existence of positive radial solutions for the following system of boundary value problems with semipositone second order elliptic equations:

where $\alpha, \beta, \gamma, \delta, k, f, g$ satisfy the conditions:

(H1) $\alpha, \beta, \gamma, \delta \geq 0$ with $\rho \equiv \gamma \beta+\alpha\gamma+\alpha \delta > 0$;

(H2) $k\in C([R_1, R_2], \mathbb R^+)$, and $k$ is not vanishing on $[R_1, R_2]$;

(H3) $f, g\in C(\mathbb R^+\times \mathbb R^+, \mathbb R)$, and there is a positive constant $M$ such that

Elliptic equations have attracted a lot of attention in the literature since they are closely related to many mathematical and physical problems, for instance, incineration theory of gases, solid state physics, electrostatic field problems, variational methods and optimal control. The existence of solutions for this type of equation in annular domains has been discussed in the literature, see for example, ^{[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]} and the references therein. In ^[1] the authors used the fixed point index to study positive solutions for the elliptic system:

with one of the following boundary conditions

In ^[2] the authors used the method of upper and lower solutions to establish the existence of positive radial solutions for the elliptic equation

where $\Omega = \left\{x \in \mathbb{R}^{N}: \ |x| < 1\right\}, N \geq 2$, and $f:[0, 1] \times \mathbb{R}^{+} \times \mathbb{R}^{+} \rightarrow \mathbb{R}$ is a continuous function.

However, we note that in most of the papers on nonlinear differential equations the nonlinear term is usually assumed to be nonnegative. In recent years boundary value problems for semipositone equations ($f(t, x) \geq-M, M > 0$) has received some attention (see ^{[19,20,21,22,23,24,25,26,27,28,29,30,31,32]}), and these equations describe and solve many natural phenomena in engineering and technical problems in real life, for example in mechanical systems, suspension bridge design, astrophysics and combustion theoretical models. In ^[19] the authors used a fixed point theorem to study the system for HIV-1 population dynamics in the fractional sense

where $D_{0+}^\alpha, D_{0+}^\beta, D_{0+}^\gamma$ are the standard Riemann-Liouville derivatives, and $f$, $g$ are two semipositone nonlinearities. In ^[28] the authors used the nonlinear alternative of Leray-Schauder type and the Guo-Krasnosel'skii fixed point theorem to study the existence of positive solutions for a system of nonlinear Riemann-Liouville fractional differential equations

where $f, g$ satisfy some superlinear or sublinear conditions:

(HZ)$_1$ There exist $M > 0$ such that $\limsup _{z \to 0} {\frac{g(t, z)}{z}} < M$ uniformly for $t \in[0, 1]$ (sublinear growth condition).

(HZ)$_2$ There exists $\left[\theta_1, \theta_2\right] \subset(0, 1)$ such that $\liminf _{z \to +\infty } \frac{f(t, z)}{z} = +\infty$ and $\liminf_{z \to +\infty} \frac{g(t, z)}{z} = +\infty$ uniformly for $t \in[\theta_1, \theta_2]$ (superlinear growth condition).

Inspired by the aforementioned work, in particular ^{[31,32,33,34]}, we study positive radial solutions for (1.1) when the nonlinearities $f, g$ satisfy the semipositone condition (H3). Moreover, some appropriate concave and convex functions are utilized to characterize coupling behaviors of our nonlinearities. Note that our conditions (H4) and (H6) (see Section 3) are more general than that in (HZ)$_1$ and (HZ)$_2$.

2.   Preliminaries

Using the methods in ^[1,4], we transform (1.1) into a system of ordinary differential equations involving Sturm-Liouville boundary conditions. Let $\varphi = \varphi(r), \phi = \phi(r), r = |z| = \sqrt{\sum\limits_{i = 1}^n z_i^2}$. Then (1.1) can be expressed by the following system of ordinary differential equations:

Then if we let $s = -\int_{r}^{R_{2}}\left(1 / t^{n-1}\right) \mathrm{d} t, t = (m-s) / m, m = -\int_{R_{1}}^{R_{2}}\left(1 / t^{n-1}\right) \mathrm{d} t$, (2.1) can be transformed into the system

where $h(t) = m^{2} r^{2(n-1)}(m(1-t)) k(r(m(1-t))).$ Consequently, (2.2) is equivalent to the following system of integral equations

where

and $\rho$ is defined in (H1).

Lemma 2.1. Suppose that (H1) holds. Then

(i)

(ii)

Proof. (i) In $G(t, s)$, we fix the second variable $s$, we have

This implies that

When $t\ge s$, we have

When $t\le s$, we have

Combining the above we obtain

(ii) In $G(t, s)$ we fix the first variable $t$, and we obtain

Thus

Lemma 2.2. Suppose that (H1) holds. Let $\vartheta(t) = G(t, t)h(t), t\in [0, 1]$. Then

where

Proof. From (H1) and Lemma 2.1(i) we have

and

Note we study (2.3) to obtain positive solutions for (1.1). However here the nonlinear terms $f, g$ can be sign-changing (see (H3)). Therefore we study the following auxiliary problem:

where $G$ is in (2.4) and $\widetilde{f}$ satisfies the condition:

(H2$'$) $\widetilde{f}\in C(\mathbb R^+, \mathbb R)$, and there exists a positive constant $M$ such that

Let $w(t) = M\int_{0}^{1} G(t, s) h(s) \mathrm{d} s, \forall t\in [0, 1]$. Then $w$ is a solution of the following boundary value problem:

Lemma 2.3. (i) If $u^*$ satisfies (2.5), then $u^*+w$ is a solution of the equation:

where

(ii) If $u^{**}$ satisfies (2.7) with $u^{**}(t)\ge w(t), t\in [0, 1]$, then $u^{**}- w$ is a positive solution for (2.5).

Proof. We omit its proof since it is immediate.

Let $E = C[0, 1]$, $\|u\| = \max_{t\in [0, 1]}|u(t)|$. Then $(E, \|\cdot\|)$ is a Banach space. Define a set on $E$ as follows:

and note $P$ is a cone on $E$. Note, $E^2 = E\times E$ is also a Banach space with the norm: $\|(u, v)\| = \|u\|+\|v\|$, and $P^2 = P\times P$ a cone on $E^2$. In order to obtain positive radial solutions for (1.1), combining with (2.5)–(2.7), we define the following operator equation:

where $A(\varphi, \phi) = (A_1, A_2)(\varphi, \phi)$, $A_i(i = 1, 2)$ are

and

Lemma 2.4. Define $P_0 = \left\{\varphi\in P: \varphi(t)\ge \frac{\rho}{(\gamma+\delta)(\beta+\alpha)}G(t, t)\|\varphi\|, t\in [0, 1]\right\}$. Then $A_i(P\times P)\subset P_0, i = 1, 2$.

Proof. We only prove it for $A_1$. If $\varphi, \phi\in P$, note the non-negativity of $\mathcal {F}_1$(denoted by $\mathcal {F}_1(\cdot, \cdot)$), from Lemma 2.1(i) we have

This implies that

Remark 2.1. (i) $w(t) = M\int_{0}^{1} G(t, s) h(s) \mathrm{d} s\in P_0$;

(ii) Note (see Corollary 1.5.1 in ^[35]):

If $k(x, y, u):\widetilde{G}\times \widetilde{G}\times \mathbb R\to \mathbb R$ is continuous ($\widetilde{G}$ is a bounded closed domain in $\mathbb R^n$), then $K$ is a completely continuous operator from $C(\widetilde{G})$ into itself, where

Note that $G(t, s), h(s), \mathcal {F}_i(i = 1, 2)$ are continuous, and also $A_i$, $A$ are completely continuous operators, $i = 1, 2$.

From Lemma 2.3 if there exists $(\varphi, \phi)\in P^2\backslash\{(0, 0)\}$ such that (2.9) holds with $(\varphi, \phi)\ge (w, w)$, then $\varphi(t), \phi(t)\ge w(t), t\in [0, 1]$, and $(\varphi-w, \phi-w)$ is a positive solution for (2.3), i.e., we obtain positive radial solutions for (1.1). Note that $\varphi, \phi\in P_0$, and from Lemma 2.1(ii) we have

Hence, if

we have $(\varphi, \phi)\ge (w, w)$. As a result, we only need to seek fixed points of (2.9), when their norms are greater than $\frac{M(\gamma+\delta)(\beta+\alpha)}{\rho}\int_{0}^{1} h(s) \mathrm{d} s$.

Let $E$ be a real Banach space. A subset $X \subset E$ is called a retract of $E$ if there exists a continuous mapping $r: E \rightarrow X$ such that $r(x) = x, \ x \in X$. Note that every cone in $E$ is a retract of $E$. Let $X$ be a retract of real Banach space $E$. Then, for every relatively bounded open subset $U$ of $X$ and every completely continuous operator $A: \overline{U} \rightarrow X$ which has no fixed points on $\partial U$, there exists an integer $i(A, U, X)$ satisfying the following conditions:

(i) Normality: $i(A, U, X) = 1$ if $A x \equiv y_0 \in U$ for any $x \in \overline{U}$.

(ii) Additivity: $i(A, U, X) = i\left(A, U_1, X\right)+i\left(A, U_2, X\right)$ whenever $U_1$ and $U_2$ are disjoint open subsets of $U$ such that $A$ has no fixed points on $\overline{U} \backslash\left(U_1 \cup U_2\right)$.

(iii) Homotopy invariance: $i(H(t, \cdot), U, X)$ is independent of $t$ $(0 \leq t \leq 1)$ whenever $H:[0, 1] \times \overline{U} \rightarrow X$ is completely continuous and $H(t, x) \neq x$ for any $(t, x) \in[0, 1] \times \partial U$.

(iv) Permanence: $i(A, U, X) = i(A, U \cap Y, Y)$ if $Y$ is a retract of $X$ and $A(\overline{U}) \subset Y$.

Then $i(A, U, X)$ is called the fixed point index of $A$ on $U$ with respect to $X$.

Lemma 2.5. (see ^[35,36]). Let $E$ be a real Banach space and $P$ a cone on $E$. Suppose that $\Omega\subset E$ is a bounded open set and that $A:\overline{\Omega}\cap P\to P$ is a continuous compact operator. If there exists $\omega_{0}\in P\backslash \{0\}$ such that

then $i(A, \Omega\cap P, P) = 0$, where $i$ denotes the fixed point index on $P$.

Lemma 2.6. (see ^[35,36]). Let $E$ be a real Banach space and $P$ a cone on $E$. Suppose that $\Omega\subset E$ is a bounded open set with $0\in\Omega$ and that $A:\overline{\Omega}\cap P\to P$ is a continuous compact operator. If

then $i(A, \Omega\cap P, P) = 1$.

3.   Main results

Denote $\mathcal {O}_{M, h} = \frac{M(\gamma+\delta)(\beta+\alpha)}{\rho}\int_{0}^{1} h(s) \mathrm{d} s$, $B_\zeta = \{u\in E: \|u\| < \zeta\}, \zeta > 0, B_\zeta^2 = B_\zeta\times B_\zeta$. We list our assumptions as follows:

(H4) There exist $p, q\in C(\mathbb R^+, \mathbb R^+)$ such that

(i) $p$ is a strictly increasing concave function on $\mathbb R^+$;

(ii) $\liminf_{v\to \infty}\frac{f(u, v)}{p(v)}\ge 1$, $\liminf_{u\to \infty}\frac{g(u, v)}{q(u)}\ge 1$;

(iii) there exists $e_1\in (\kappa_1^{-2}, \infty)$ such that $\liminf_{z\to\infty}\frac{p(\mathcal {L}_{G, h}q(z))}{z}\ge e_1\mathcal {L}_{G, h}$, where $\mathcal {L}_{G, h} = \max_{t, s\in [0, 1]}G(t, s)h(s)$.

(H5) There exists $Q_i\in \left(0, \frac{\mathcal {O}_{M, h}}{\kappa_2}\right)$ such that

(H6) There exist $\zeta, \eta\in C(\mathbb R^+, \mathbb R^+)$ such that

(i) $\zeta$ is a strictly increasing convex function on $\mathbb R^+$;

(ii) $\limsup_{v\to \infty}\frac{f(u, v)}{\zeta(v)}\le 1$, $\limsup_{u\to \infty}\frac{g(u, v)}{\eta(u)}\le 1$;

(iii) there exists $e_2\in (0, \kappa_2^{-2})$ such that $\limsup_{z\to\infty}\frac{\zeta(\mathcal {L}_{G, h}\eta(z))}{z}\le e_2\mathcal {L}_{G, h}$.

(H7) There exists $\widetilde{Q}_i\in \left(\frac{\mathcal {O}_{M, h}}{\kappa_2\mathcal L_G}, \infty\right)$ such that

where $\mathcal L_G = \max_{t\in [0, 1]} \frac{\rho}{(\gamma+\delta)(\beta+\alpha)}G(t, t)$.

Remark 3.1. Condition (H4) implies that $f$ grows $p(v)$-superlinearly at $\infty$ uniformly on $u\in \mathbb R^+$, $g$ grows $q(u)$-superlinearly at $\infty$ uniformly on $v\in \mathbb R^+$; condition (H6) implies that $f$ grows $\zeta(v)$-sublinearly at $\infty$ uniformly on $u\in \mathbb R^+$, $g$ grows $\eta(u)$-sublinearly at $\infty$ uniformly on $v\in \mathbb R^+$.

Theorem 3.1. Suppose that (H1)–(H5) hold. Then (1.1) has at least one positive radial solution.

Proof. Step 1. When $\varphi, \phi\in \partial B_{\mathcal {O}_{M, h}}\cap P$, we have

Suppose the contrary i.e., if (3.1) is false, then there exist $\varphi_0, \phi_0\in \partial B_{\mathcal {O}_{M, h}}\cap P$ and $\lambda_0\in [0, 1]$ such that

This implies that

and

From (H5) we have

Thus

which contradicts (3.3), and thus (3.1) holds. From Lemma 2.6 we have

Step 2. There exists a sufficiently large $R > \mathcal {O}_{M, h}$ such that

where $\varrho_1\in P_0$ is a given element. Suppose the contrary. Then there are $\varphi_1, \phi_1\in \partial B_R \cap P, \lambda_1\ge 0$ such that

This implies that

From Lemma 2.4 and $\varrho_1\in P_0$ we have

Note that $\|\varphi_1\| = \|\phi_1\| = R > \mathcal {O}_{M, h}$, and thus $\varphi_1(t)\ge w(t), \phi_1(t)\ge w(t), t\in [0, 1]$.

By (H4)(ii) we obtain

This implies that there exist $c_1, c_2 > 0$ such that

Therefore, we have

and

Consequently, we have

From (H4)(iii), there is a $c_3 > 0$ such that

Combining with (H4)(i), we have

Substituting this inequality into (3.8) we have

Multiply by $\vartheta(t)$ on both sides of the above and integrate over $[0, 1]$ and use Lemma 2.2 to obtain

From this inequality we have

and thus

Note that (3.7), $\varphi_1\in P_0$, and we have

On the other hand, multiply by $\vartheta(t)$ on both sides of (3.8) and integrate over $[0, 1]$ and use Lemma 2.2 to obtain

From Remark 2.1 we have $w\in P_0$, note that $\|\phi_1\| = R > \frac{M(\gamma+\delta)(\beta+\alpha)}{\rho}\int_{0}^{1} h(s) \mathrm{d} s\ge \|w\|$ and $\phi_1\in P_0$, then $\phi_1-w\in P_0$. By the concavity of $p$ we have

This implies that

From (H4)(i) we have

Therefore, there exists $\mathcal {O}_{\phi_1} > 0$ such that $\|\phi_1\|\le\mathcal {O}_{\phi_1}$.

We have prove the boundedness of $\varphi_1, \phi_1$ when (3.6) holds, i.e., when $\varphi_1, \phi_1\in \partial B_R\cap P$, there exist a positive constant to control the norms of $\varphi_1, \phi_1$. Now we choose a sufficiently large

Then when $\varphi_1, \phi_1\in \partial B_{R_1} \cap P$, (3.6) is not satisfied, and thus (3.5) holds. From Lemma 2.5 we have

Combining (3.4) with (3.10) we have

Then the operator $A$ has at least one fixed point (denoted by $(\varphi^*, \phi^*)$) on $(B^2_{R_1}\backslash \overline{B}^2_{\mathcal {O}_{M, h}})\cap P^2$ with $\varphi^*(t), \phi^*(t)\ge w(t), t\in [0, 1]$. Therefore, $(\varphi^*-w, \phi^*-w)$ is a positive solution for (2.2), and (1.1) has at least one positive radial solution.

Theorem 3.2. Suppose that (H1)–(H3), (H6) and (H7) hold. Then (1.1) has at least one positive radial solution.

Proof. Step 1. When $\varphi, \phi\in \partial B_{\mathcal {O}_{M, h}}\cap P$, we have

where $\varrho_2\in P$ is a given element. Suppose the contrary. Then there exist $\varphi_2, \phi_2\in \partial B_{\mathcal {O}_{M, h}}\cap P, \lambda_2\ge 0$ such that

This implies that

Then we have

From (H7) we have

By the condition on $\widetilde{Q}_i$ we have

and this contradicts (3.12), so (3.11) holds. By Lemma 2.5 we have

Step 2. There exists a sufficiently large $R > \mathcal {O}_{M, h}$ such that

Suppose the contrary. Then there exist $\varphi_3, \phi_3\in \partial B_R \cap P, \lambda_3\in [0, 1]$ such that

Combining with Lemma 2.4 we have

Note that $\varphi_3, \phi_3\in \partial B_R\cap P,$ and then $\varphi_3(t)-w(t), \phi_3(t)-w(t)\ge 0, t\in [0, 1]$. Hence, from (H6) we have

This implies that there exists $\widetilde{M} > 0$ such that

By similar methods as in Theorem 3.1, choosing $R > \widetilde{M}$, and from (3.15) we obtain

and

From (H6)(iii), there exists $c_4 > 0$ such that

By the convexity of $\zeta$ we have

Substituting this inequality into (3.18) we have

Consequently, we have

Multiply by $\vartheta(t)$ on both sides of (3.22) and integrate over $[0, 1]$ and use Lemma 2.2 to obtain

and we have

Note that (3.16), $w\in P_0$, and

By the triangle inequality we have

On the other hand, from (3.20) we have

Note that $\frac{c_4e_2 \kappa_2^3}{1-e_2 \kappa_2^2}+c_4\kappa_2$ is independent to $R$, and using (H6)(i) there exists $\mathcal {O}_{\phi_3} > 0$ such that

and then

Therefore, when $\varphi_3, \phi_3\in \partial B_R \cap P$, we obtain there is a positive constant to control the norms of $\varphi_3, \phi_3$. Then if we choose

then (3.14) holds, and from Lemma 2.6 we have

From (3.13) and (3.23) we have

Then the operator $A$ has at least one fixed point (denoted by $(u^{**}, v^{**})$) on $(B^2_{R_2}\backslash \overline{B}^2_{\mathcal {O}_{M, h}})\cap P^2$ with $u^{**}(t), v^{**}(t)\ge w(t), t\in [0, 1]$. Therefore, $(u^{**}-w, v^{**}-w)$ is a positive solution for (2.2), and (1.1) has at least one positive radial solution.

We now provide some examples to illustrate our main results. Let $\alpha = \beta = \gamma = \delta = 1,$ and $k(|z|) = e^{|z|}, z\in \mathbb R^n$. Then (H1) and (H2) hold.

Example 3.1. Let $p(\phi) = \phi^{\frac{4}{5}}, q(\varphi) = \varphi^2, \varphi, \phi\in \mathbb R^+$. Then $\liminf_{z\to\infty}\frac{p(\mathcal {L}_{G, h}q(z))}{z} = \liminf_{z\to\infty}\frac{\mathcal {L}^{\frac{4}{5}}_{G, h}z^\frac{8}{5}}{z}\ge \infty$, and (H4)(i), (iii) hold. If we choose

then (H3) holds, and when $\varphi, \phi\in [0, \mathcal {O}_{M, h}]$, we have

Hence, (H5) holds. Also we have

Then (H4)(ii) holds. As a result, all the conditions in Theorem 3.1 hold, and (1.1) has at least one positive radial solution.

Example 3.2. Let $\zeta(\phi) = \phi^2, \eta(\varphi) = \varphi^{\frac{2}{5}}$, $\varphi, \phi\in \mathbb R^+$. Then $\limsup_{z\to\infty}\frac{\zeta(\mathcal {L}_{G, h}\eta(z))}{z} = \limsup_{z\to\infty}\frac{\mathcal {L}^2_{G, h}z^{\frac{4}{5}}}{z}$ = $0\le e_2\mathcal {L}_{G, h}$, and (H7)(i), (iii) hold. If we choose

where $\alpha_{1} \in(0, 2), \alpha_{2} \in\left(0, \frac{2}{5}\right)$. Then (H3) holds. Moreover, we have

and

Therefore, $(\mathrm{H} 6)$ and $(\mathrm{H} 7)$ (ii) hold. As a result, all the conditions in Theorem 3.2 hold, and (1.1) has at least one positive radial solution.

Remark 3.2. Note that condition $(HZ)_2$ is often used to study various kinds of semipositone boundary value problems (for example, see ^{[19,22,23,26,28,29,30]}). However, in Example 3.1 we have

Comparing with $(HZ)_2$ we see that our theory gives new results for boundary value problem with semipositone nonlinearities.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (12101086), Changzhou Science and Technology Planning Project (CJ20210133), Natural Science Foundation of Chongqing (cstc2020jcyj-msxmX0123), and Technology Research Foundation of Chongqing Educational Committee (KJQN202000528).

Conflict of interest

The authors declare no conflict of interest.