This article proposes a locomotion controller inspired by black Knifefish for undulating elongated fin robot. The proposed controller is built by a modified CPG network using sixteen coupled Hopf oscillators with the feedback of the angle of each fin-ray. The convergence rate of the modified CPG network is optimized by a reinforcement learning algorithm. By employing the proposed controller, the undulating elongated fin robot can realize swimming pattern transformations naturally. Additionally, the proposed controller enables the configuration of the swimming pattern parameters known as the amplitude envelope, the oscillatory frequency to perform various swimming patterns. The implementation processing of the reinforcement learning-based optimization is discussed. The simulation and experimental results show the capability and effectiveness of the proposed controller through the performance of several swimming patterns in the varying oscillatory frequency and the amplitude envelope of each fin-ray.
Citation: Van Dong Nguyen, Dinh Quoc Vo, Van Tu Duong, Huy Hung Nguyen, Tan Tien Nguyen. Reinforcement learning-based optimization of locomotion controller using multiple coupled CPG oscillators for elongated undulating fin propulsion[J]. Mathematical Biosciences and Engineering, 2022, 19(1): 738-758. doi: 10.3934/mbe.2022033
[1] | Michael Eden, Michael Böhm . Homogenization of a poro-elasticity model coupled with diffusive transport and a first order reaction for concrete. Networks and Heterogeneous Media, 2014, 9(4): 599-615. doi: 10.3934/nhm.2014.9.599 |
[2] | Tasnim Fatima, Ekeoma Ijioma, Toshiyuki Ogawa, Adrian Muntean . Homogenization and dimension reduction of filtration combustion in heterogeneous thin layers. Networks and Heterogeneous Media, 2014, 9(4): 709-737. doi: 10.3934/nhm.2014.9.709 |
[3] | Alexander Mielke, Sina Reichelt, Marita Thomas . Two-scale homogenization of nonlinear reaction-diffusion systems with slow diffusion. Networks and Heterogeneous Media, 2014, 9(2): 353-382. doi: 10.3934/nhm.2014.9.353 |
[4] | Hirofumi Notsu, Masato Kimura . Symmetry and positive definiteness of the tensor-valued spring constant derived from P1-FEM for the equations of linear elasticity. Networks and Heterogeneous Media, 2014, 9(4): 617-634. doi: 10.3934/nhm.2014.9.617 |
[5] | Catherine Choquet, Ali Sili . Homogenization of a model of displacement with unbounded viscosity. Networks and Heterogeneous Media, 2009, 4(4): 649-666. doi: 10.3934/nhm.2009.4.649 |
[6] | Steinar Evje, Aksel Hiorth, Merete V. Madland, Reidar I. Korsnes . A mathematical model relevant for weakening of chalk reservoirs due to chemical reactions. Networks and Heterogeneous Media, 2009, 4(4): 755-788. doi: 10.3934/nhm.2009.4.755 |
[7] | Oleh Krehel, Toyohiko Aiki, Adrian Muntean . Homogenization of a thermo-diffusion system with Smoluchowski interactions. Networks and Heterogeneous Media, 2014, 9(4): 739-762. doi: 10.3934/nhm.2014.9.739 |
[8] | Brahim Amaziane, Leonid Pankratov, Andrey Piatnitski . An improved homogenization result for immiscible compressible two-phase flow in porous media. Networks and Heterogeneous Media, 2017, 12(1): 147-171. doi: 10.3934/nhm.2017006 |
[9] | Jean-Marc Hérard, Olivier Hurisse . Some attempts to couple distinct fluid models. Networks and Heterogeneous Media, 2010, 5(3): 649-660. doi: 10.3934/nhm.2010.5.649 |
[10] | Maksym Berezhnyi, Evgen Khruslov . Non-standard dynamics of elastic composites. Networks and Heterogeneous Media, 2011, 6(1): 89-109. doi: 10.3934/nhm.2011.6.89 |
This article proposes a locomotion controller inspired by black Knifefish for undulating elongated fin robot. The proposed controller is built by a modified CPG network using sixteen coupled Hopf oscillators with the feedback of the angle of each fin-ray. The convergence rate of the modified CPG network is optimized by a reinforcement learning algorithm. By employing the proposed controller, the undulating elongated fin robot can realize swimming pattern transformations naturally. Additionally, the proposed controller enables the configuration of the swimming pattern parameters known as the amplitude envelope, the oscillatory frequency to perform various swimming patterns. The implementation processing of the reinforcement learning-based optimization is discussed. The simulation and experimental results show the capability and effectiveness of the proposed controller through the performance of several swimming patterns in the varying oscillatory frequency and the amplitude envelope of each fin-ray.
An important goal of system biology is to understand the mechanisms that govern the protein regulatory dynamics. The so-called feedback loop is believed to be widely present in various eukaryotic cellular processes and regulates the gene expression of a broad class of intracellular proteins. It is usually understood that in the feedback loop transcription factors are regulatory proteins which activate transcription in eukaryotic cells; they act by binding to a specific DNA sequences in the nucleus, either activating or inhibiting the binding of RNA polymerase to DNA; mRNA is transcribed in the nucleus and is in turn translated in the cytoplasm. Such a feedback loop was described by systems of ordinary differential equations in [6, 7, 8,17,21].
Many scholars have noticed that the genetic regulatory dynamics are dependent on the intracellular transport of mRNA and protein [13,18]. In particular, models of genetic regulatory dynamics are developed to explain a variety of sustained periodic biological phenomena [19]. Hence the possible roles played by the time delays in the signaling pathways become an interesting research topic and we are interested how time delays are associated with periodic oscillations. Regulatory models with constant delays are investigated in the work of [18,24], among many others, where time delay is used as a parameter and is tuned to observe stable periodic oscillations. It was pointed out in [19] that if the simplification of constant time delay is too drastic, another approach is to assume a distributed time delay. In the work of Busenburg and Mahaffy [2], both diffusive macromolecular transport represented as a distributed delay and active macromolecular transport modeled as time delay are considered. It was demonstrated that stable periodic oscillations can occur when transport was slowed by increasing the time delay, and that oscillations were dampened if transport was slowed by reducing the diffusion coefficients.
We follow the idea of the work of [2] to consider both diffusion transport and active transport in genetic regulatory dynamics, while we start from the Goodwin's model [6] in contrast to the reaction-diffusion equations in [2]. Namely, we will provide a new perspective to investigate genetic regulatory dynamics which generalizes the previous extension of Goodwin's model with constant delay and simplifies the approach adopted by [2]. To be more specific, we begin with the prototype model for Hes1 system (see Section 2 for a brief introduction) and discuss how to model the diffusion time if we consider the effect of the fluctuation of concentrations of the substances in the Hes1 system. We show that modeling the diffusion time will lead to a model with threshold type distributed delay, which is also called threshold type state-dependent delay [1,16]. With a general model with threshold type state-dependent delay, we are interested how the model with state-dependent delay differs from those with only constant delays, how the stabilities of the steady state and the periodic oscillations depend on the state-dependent delay.
We organize the remaining of the paper as follows. In Section 2, we model the diffusion time in regulatory dynamics and set up a prototype system of differential equations with state-dependent delay for the Hes1 system. In Section 3, we consider a general model with state-dependent delay and conduct linear stability analysis of the equivalent model and conduct a local Hopf bifurcation analysis of the system using the multiple time scale method. Numerical simulations on the model of Hes1 system with state-dependent delay will be illustrated in Section 4. We conclude the investigation at Section 5.
In this section, we motivate the modeling of diffusion time delay with the Hes1 regulatory system. Hes1 is a protein which belongs to the basic helix-loop-helix (bHLH, a protein structural motif) family of transcription factors. It is a transcriptional repressor of genes that require a bHLH protein for their transcription. As a member of the bHLH family, it is a transcriptional repressor that influences cell proliferation and differentiation in embryogenesis. Hes1 regulates its own expression via a negative feedback loop, and oscillates with approximately 2-hour periodicity [9]. However, the molecular mechanism of such oscillation remains to be determined [9]. Mathematical modeling of Hes1 regulatory system has been an active research area in the last decade. See among many others, [12,20].
The basic reaction kinetics for the regulatory system (Figure 1) can be described by the following ordinary differential equations which is in general called the Goodwin model (See [7,8]). Denote by
{dxm(t)dt=−μmxm(t)+αm1+(yp(t)ˉy)h,dyp(t)dt=−μpyp(t)+αpxm(t), | (1) |
where
Since mRNA is typically transcribed in nucleus and then translated to cytoplasm for protein synthesis, there exists transcriptional and translational delays in the real reaction kinetics. For a better description of the intracellular processes, (1) was extended into the following model of delay differential equation (see, e. g., [12])
{dxm(t)dt=−μmxm(t)+αm1+(yp(t−τm)ˉy)h,dyp(t)dt=−μpyp(t)+αpxm(t−τp), | (2) |
where
For simplicity, we combine the reaction or transcription time with the translation time between the nucleus and the cytoplasm and call the combination is the diffusion time which will be denoted by
(A1)
Now we drop the subscripts for the variables
The next problem is how
T=−L(x(t)−x(t−τ))2J. | (3) |
Then we have
(A2) there exist constants
Namely, we assume that the diffusion time is linearly dependent on the fluctuation of the concentrations of mDNA. We remark that
Now we obtain the following state-dependent delay differential equations,
{dx(t)dt=−μmx(t)+αm1+(y(t−τ)ˉy)h,dy(t)dt=−μpy(t)+αpx(t−τ),τ(t)=c(x(t)−x(t−τ))+ϵ. | (4) |
which provides another view on the models (1) and (2): If we assume that the transcription and translation of the newly produced solute is instant, namely,
It was shown in [10] that system with the state-dependent delay in the form of
Regarding system (4) as a prototype model of a range of genetic regulatory dynamics, we consider the following system,
{dx(t)dt=−μmx(t)+f(y(t−τ)),dy(t)dt=−μpy(t)+g(x(t−τ)),τ(t)=ϵ+c(x(t)−x(t−τ)), | (5) |
where
B1) System (5) has at least one equilibrium
{x′(0)=−μmx(0)+f(y(−τ0)),y′(0)=−μpy(0)+g(x(−τ0)),τ0=ϵ+c(x(0)−x(−τ0)), | (6) |
system (5) has a unique solution
B2) Let
We rewrite the equation for the delay in (4) as
∫tt−τ(t)1−c˙x(s)ϵds=1. | (7) |
Let
η=∫t01−cu(s)ϵds,r(η)=x(t),ξ(η)=y(t),k(η)=τ(t). | (8) |
Then we have
{η−1=∫t−τ(t)01−cu(s)ϵds,r(η−1)=x(t−τ(t)),ξ(η−1)=y(t−τ(t)),k(η)=ϵ+c(r(η)−r(η−1)). | (9) |
Note that we have
{˙x(t)=drdη⋅dηdt=˙r(η)⋅1−c˙xϵ,˙y(t)=dξdη⋅dηdt=˙ξ(η)⋅1−c˙xϵ, | (10) |
Then by (9) and (10), system (4) is transformed into
{drdη=ϵ−μmr(η)+f(ξ(η−1))1−c(−μmr(η)+f(ξ(η−1))),dξdη=ϵ−μpξ(η)+g(r(η−1))1−c(−μmr(η)+f(ξ(η−1))),k(η)=ϵ+c(r(η)−r(η−1)), | (11) |
where the assumption that
{1ϵdrdη=−μmr(η)+f(ξ(η−1))1ϵdξdη=−μpξ(η)+g(r(η−1))k(η)=ϵ, | (12) |
which is equivalent to (5) with
{dx(t)dt=−μmx(t)+f(y(t−τ)),dy(t)dt=−μpy(t)+g(x(t−τ)),τ(t)=ϵ, | (13) |
in the sense of the time-domain transformations leading to system (11).
Comparing system (11) with system (12), we see that both systems have the same set of equilibria. We also observe that, if
Now we study the stability of the equilibrium of system (11). Note that the variable
{−μmr∗+f(ξ∗))=0,−μpξ∗+g(r∗)=0. |
We translate the equilibrium of (11) to the origin by letting
F(v(η−1))=f(v(η−1)+ξ∗)−f(ξ∗),G(u(η−1))=g(u(η−1)+r∗)−g(r∗). |
Then we have
{1ϵdudη=−μmu+F(v(η−1))1−c[−μmu+F(v(η−1))],1ϵdvdη=−μpv+G(u(η−1))1−c[−μmu+F(v(η−1))]. | (14) |
Linearization of (14) at
˙x(η)=ϵMx(η)+ϵNx(η−1), | (15) |
where
(λ+ϵμm)(λ+ϵμp)−ϵ2f′(ξ∗)g′(r∗)e−2λ=0. | (16) |
Let
z2+τ(μm+μp)z+τ2μmμp+τ2he−z=0, | (17) |
which is the same equation investigated in [3]. Then by the results in [3] we have,
Lemma 3.1. Consider the characteristic equation (16), where
β2−ϵ2(μmμp)=ϵ(μm+μp)βcot2β, |
has a unique solution for
ⅰ) If
ⅱ) If
Lemma 3.2. Consider the characteristic equation (16), where
μmμp<−f′(ξ∗)g′(r∗), |
then there exists a unique
(μm+μp)β(ϵ0)+ϵ0f′(ξ∗)g′(r∗)sin2β(ϵ0)=0, |
and
tan2β(ϵ0)=√l(μm+μp)1−lμmμp, |
where
l=μ2m+μ2p+√(μ2m−μ2p)2+4f′2(ξ∗)g′2(r∗)2(f′2(ξ∗)g′2(r∗)−μ2mμ2p),ϵ0=√lβ(ϵ0). |
Proof. By Lemma 3.1
{β2(ϵ0)−ϵ02(μmμp)−ϵ0(μm+μp)β(ϵ0)cot2β(ϵ0)=0,(μm+μp)β(ϵ0)+ϵ0f′(ξ∗)g′(r∗)sin2β(ϵ0)=0. | (18) |
Eliminating the trigonometric terms in (18), we obtain that
ϵ40(μ2mμ2p−f′2(ξ∗)g′2(r∗)))+ϵ20(μ2m+μ2p)β2(ϵ0)+β4(ϵ0)=0, |
which can be regarded as a quadratic equation of
ϵ20=lβ2(ϵ0),l=μ2m+μ2p+√(μ2m−μ2p)2+4f′2(ξ∗)g′2(r∗)2(f′2(ξ∗)g′2(r∗)−μ2mμ2p). |
Bringing
tan2β(ϵ0)=√l(μm+μp)1−lμmμp. |
In this section we use multiple time scale method to find the normal form of system (11) at the first critical value
{α2−β2+ϵ(μm+μp)α+ϵ2μmμp−ϵ2f′(ξ∗)g′(r∗)e−2αcos2β=0,2αβ+ϵ(μm+μp)β+ϵ2f′(ξ∗)g′(r∗)e−2αsin2β=0. | (19) |
By Implicit function theorem, we can show that
{dαdϵ(ϵ∗(μm+μp)+2ϵ∗2f′(ξ∗)g′(r∗)cos2β)+dβdϵ(−2β+2ϵ∗2f′(ξ∗)g′(r∗)sin2β)=−2ϵ∗μmμp+2f′(ξ∗)g′(r∗)cos2β,dαdϵ(2β−2ϵ∗2f′(ξ∗)g′(r∗)sin2β)+dβdϵ(ϵ∗(μm+μp)+2ϵ∗2f′(ξ∗)g′(r∗)cos2β)=−(μm+μp)β−2ϵ∗f′(ξ∗)g′(r∗)sin2β. | (20) |
Noticing that if
{ϵ∗2f′(ξ∗)g′(r∗)cos2β=ϵ∗2μmμp−β2,ϵ∗2f′(ξ∗)g′(r∗)sin2β=−ϵ∗(μm+μp)β. | (21) |
Combining (20) and (21), we obtain that
dαdϵ|ϵ=ϵ∗,λ=±iβ=2β2ϵ∗(ϵ∗2(μ2m+μ2p)+2β2)(ϵ∗(μm+μp)+2ϵ∗2μmμp−2β2)2+(2β+2βϵ∗(μm+μp))2>0. | (22) |
To illustrate the stability of system (5), we have,
Theorem 3.3. Consider system (5) with an equilibrium
ⅰ) If
ii) If
Proof. By the technique of formal linearization [4,23], we know that the equilibrium stability of system (5) is equivalent to that of system (13) which has the same equilibrium and characteristic equation for system (11). Then by Lemma 3.1, the stability of
Let
1ϵ˙x(η)=[S((x(η),x(η−1))H((x(η),x(η−1))], |
where
{1ϵdudη=−μmu(η)+f′(ξ∗)v(η−1)+cμ2mu2(η)−2cμmf′(ξ∗)u(η)v(η−1)+(12f″ | (23) |
In this section we compute the normal form of system (14) using its Taylor expansion at (23). We seek a uniform second-order approximate solution in the form
\begin{align}\label{mts-1} x(\eta;\, s) = sx_1(T_0, \, T_2)+s^2x_2(T_0, \, T_2)+s^3x_3(T_0, \, T_2)+\cdots, \end{align} | (24) |
where
\begin{align}\label{mts-2} \frac{\mathrm{d}}{\mathrm{d}\eta} = \frac{\partial}{\partial T_0}+s^2\frac{\partial }{\partial T_2}+\cdots = D_0+s^2 D_2+\cdots. \end{align} | (25) |
By (24) and (25) we have
\begin{align}\label{mts-3} x(\eta-1)& = x(T_0-1, \, T_2-s^2) = \sum\limits_{n = 1}^\infty s^n x_n(T_0-1, \, T_2)-s^3 D_2 x_1(T_0-1, \, T_2)\cdots. \end{align} | (26) |
Let
\begin{align*} &\frac{1}{(\epsilon^*+s^2\delta)}(D_0+s^2 D_2)\sum^{\infty}_{n = 1}s^n x_n(T_0, \, T_2) \\ = &\frac{1}{(\epsilon^*+s^2\delta)}\left[s D_0 x_1(T_0, \, T_2)+s^2D_0x_2(T_0, \, T_2)\right.\\ & \left.+ s^3D_0 x_3(T_0, \, T_2)+ s^3 D_2 x_1(T_0, \, T_2)+s^4 D_2 x_2(T_0, \, T_2)+\cdots\right], \end{align*} |
and the right hand side becomes
\begin{align*} M\sum^{\infty}_{n = 1}s^n x_n (T_0, \, T_2) +N\left(\sum\limits_{n = 1}^\infty s^n x_n(T_0-1, \, T_2)-s^3D_2 x_1(T_0-1, \, T_2)+\cdots\right)+\cdots. \end{align*} |
Comparing the like powers of
\begin{align} &D_0 x_1-\epsilon^*Mx_1-\epsilon^*Nx_1(T_0-1, \, T_2) = 0, \label{s-eqn-1} \end{align} | (27) |
![]() |
(28) |
![]() |
(29) |
We know that when
x_1 = A(T_2)\theta e^{iw^*T_0}+\bar{A}(T_2)\bar{\theta}e^{-iw^*T_0}, |
where
\begin{align*} \left(iw^*I+\epsilon^*\begin{bmatrix} \mu_m&0\\ 0&\mu_p \end{bmatrix} -\epsilon^*\begin{bmatrix} 0&f'(\xi^*)e^{-iw^*}\\ g'(r^*)e^{-iw^*}&0 \end{bmatrix}\right) \left(\begin{matrix}\theta_1\\ \theta_2 \end{matrix}\right)& = 0. \end{align*} |
Leting
\begin{align*} \theta = \left[ \begin{matrix} 1\\ \frac{e^{i w^*}(i w^*+\epsilon^*\mu_m)}{\epsilon^* f'(\xi^*)} \end{matrix} \right]. \end{align*} |
Substitute
![]() |
(30) |
Equation (28) has a particular solution of the form
x_2 = aA^2e^{i2w^*T_0}+bA\bar{A}+\bar{a}\bar{A}^2e^{-i2 w^*T_0}, |
where
\begin{align}\label{eqn-5-12} &(aA^2e^{i2w^*T_0}\cdot i2w^*+\bar{a}\bar{A}^2e^{-i2w^*T_0}(-i2w^*))\\&+\epsilon^*\left[\begin{matrix} \mu_m&0\notag\\ 0&\mu_p \end{matrix}\right] (aA^2e^{i2w^*T_0}+bA\bar{A}+\bar{a}\bar{A}^2e^{-i2w^*T_0})\notag\\ &-\epsilon^*\left[\begin{matrix} 0&f'(\xi^*)\\ g'(r^*) &0 \end{matrix}\right] (aA^2e^{i2w^*(T_0-1)}+bA \bar{A}+\bar{a} \bar{A}^2 e^{-i2w^*(T_0-1)})\notag \end{align} | (31) |
![]() |
(32) |
Equating the coefficients of the terms with
\begin{align}\label{eqn-5-13} \left\{ \begin{aligned} &(i2w^*)A^2a_1+\epsilon^*\mu_mA^2a_1-\epsilon^*f'(\xi^*)A^2e^{-i2w^*}a_2 \\ = & \epsilon^*c\mu_m^2A^2-2\epsilon^*c\mu_mf'(\xi^*)A^2\theta_2e^{-iw^*} +\frac{\epsilon^*}{2}(f''(\xi^*)+2cf'^2(\xi^*))A^2\theta_2^2e^{-i2w^*}, \\&(i2w^*)A^2a_2+\epsilon^*\mu_pA^2a_2-\epsilon^*g'(r^*)A^2e^{-i2w^*}a_1 \\ = & -\epsilon^*c\mu_pf'(\xi^*)\theta^2_2A^2e^{-iw^*}+\epsilon^*c\mu_m\mu_pA^2\theta_2 +\frac{\epsilon^*}{2}g''(r^*)A^2e^{-2iw^*}\\ &+\epsilon^*cf'(\xi^*)g'(r^*)A^2\theta_2e^{-2iw^*} -\epsilon^*c\mu_mg'(r^*)A^2e^{-iw^*}. \end{aligned} \right. \end{align} | (33) |
Equation (33) has a solution for
(B3)
We have
\begin{align}\label{eqn-b-1-2} \left\{ \begin{aligned} a_1 = &\frac{\epsilon^*}{ (i2w^*+\epsilon^*\mu_m)(i2w^*+\epsilon^*\mu_p) -{\epsilon^*}^2f'(\xi^*)g'(r^*)e^{-i4w^*} }\\ &\times\bigg[\left( (c\mu_m^2-2c\mu_mf'(\xi^*)\theta_2e^{-iw^*} +\frac{1}{2}(f''(\xi^*)+2cf'^2(\xi^*))\theta_2^2e^{-i2w^*} \right) \\ & \times (i2w^*+\epsilon^*\mu_p) +\bigg(-c\mu_pf'(\xi^*)\theta^2_2 e^{-iw^*}+ c\mu_m\mu_p \theta_2 +\frac{1}{2}g''(r^*) e^{-2iw^*}\\ & + cf'(\xi^*)g'(r^*) \theta_2e^{-2iw^*}-c\mu_mg'(r^*) e^{-iw^*}\bigg)(\epsilon^*f'(\xi^*)e^{-i2w^*}) \bigg], \\ a_2 = &\frac{\epsilon^*}{ (i2w^*+\epsilon^*\mu_m)(i2w^*+\epsilon^*\mu_p) -{\epsilon^*}^2f'(\xi^*)g'(r^*)e^{-i4w^*} }\\ &\times\bigg[\left( (c\mu_m^2-2c\mu_mf'(\xi^*)\theta_2e^{-iw^*} +\frac{1}{2}(f''(\xi^*)+2cf'^2(\xi^*))\theta_2^2e^{-i2w^*} \right) \\ & \times (\epsilon^*g'(r^*)e^{-i2w^*} ) +\bigg(-c\mu_pf'(\xi^*)\theta^2_2 e^{-iw^*}+ c\mu_m\mu_p \theta_2 +\frac{1}{2}g''(r^*) e^{-2iw^*}\\ & + cf'(\xi^*)g'(r^*) \theta_2e^{-2iw^*}-c\mu_mg'(r^*) e^{-iw^*}\bigg)(i2w^*+\epsilon^*\mu_m) \bigg]. \end{aligned} \right. \end{align} | (34) |
Equating the coefficients of the
\begin{align*} \left\{ \begin{aligned} \epsilon^*\mu_mb_1-\epsilon^*f'(\xi^*)b_2 = &\, \, 2\epsilon^*c\mu_m^2-2\epsilon^*c\mu_mf'(\xi^*)(\bar{\theta_2}e^{iw^*}+\theta_2e^{-iw^*})\\ &\, \, +\epsilon^*(f''(\xi^*)+2cf'^2(\xi^*))\theta_2\bar{\theta_2}, \\ -\epsilon^*g'(r^*)b_1+\epsilon^*\mu_pb_2 = &\, \, -\epsilon^*c\mu_pf'(\xi^*)\theta_2\bar{\theta_2}(e^{iw^*}+e^{-iw^*})+\epsilon^*c\mu_m\mu_p(\theta_2+\bar{\theta_2})\\ &+\epsilon^*g''(r^*)+\epsilon^*cf'(\xi^*)g'(r^*)(\theta_2+\bar{\theta_2})-\epsilon^*c\mu_mg'(r^*)\\ &\times(e^{iw^*}+e^{-iw^*}). \end{aligned} \right. \end{align*} |
Noticing that
\begin{align*} \left\{ \begin{aligned} b_1 = &\frac{1}{{\epsilon^*}^2f'^2(\xi^*)(\mu_m\mu_p-f'(\xi^*)g'(r^*))} \bigg[ \mu_pf''(\xi^*){w^*}^2+\mu_pf''(\xi^*){\epsilon^*}^2\mu_m^2\\ &+2f'^2(\xi^*)c\mu_p{w^*}^2-2f'^2(\xi^*)c\mu_p{w^*}^2\cos w^*-2c(f'^3(\xi^*)g'(r^*)\\ &+\mu_m\mu_pf'^2(\xi^*)){\epsilon^*}{w^*}\sin w^* +f'^3(\xi^*)g''(r^*){\epsilon^*}^2 \bigg], \\ b_2 = &\frac{1}{{\epsilon^*}^2f'^2(\xi^*)(\mu_m\mu_p-f'(\xi^*)g'(r^*))} \bigg[ \mu_mf'^2(\xi^*)g''(r^*){\epsilon^*}^2+ f''(\xi^*)g'(r^*){w^*}^2\\ & +f''(\xi^*)g'(r^*)\mu_m^2{\epsilon^*}^2+2cf'^2(\xi^*)g'(r^*){w^*}^2-2c\mu_m\mu_pf'(\xi^*){w^*}^2\cos w^*\\ &-2c(\mu_mf'^2(\xi^*)g'(r^*)+\mu_m^2\mu_pf'(\xi^*)){\epsilon^*}{w^*}\sin w^* \bigg]. \end{aligned} \right. \end{align*} |
Substituting the expressions of
![]() |
(35) |
Note that
\begin{align}\label{eqn-s-515} \left[iw^*I-\epsilon^*M-\epsilon^*Ne^{-iw^*}\right]\phi = \chi. \end{align} | (36) |
where
\begin{align*} \left[-iw^*I-\epsilon^*M^T-\epsilon^*N^Te^{iw^*}\right]d = 0, \end{align*} |
we require
\begin{align}\label{eqn-s-516} \bar{d}^T\chi = 0, \end{align} | (37) |
which is an ordinary differential equation of
\begin{align*} d = \frac{1}{-2iw^*+\epsilon^*(\mu_m+\mu_p)} \begin{bmatrix} {-iw^*+\epsilon^*\mu_p}\\ \epsilon^*e^{iw^*}f'(\xi^*) \end{bmatrix}. \end{align*} |
Bringing the expressions of
![]() |
(38) |
By (37) and (38) and noticing that
iw^*\theta-\epsilon^*M\theta-\epsilon^*N\theta e^{-iw^*} = 0, |
we have the following normal form:
![]() |
(39) |
We arrive at,
Theorem 3.4. Assume that (B1), (B2) and (B3) hold. The normal form of system (5) near its equilibrium is (39).
We remark that if the state-dependent translation time
\begin{array}{l} A' = \frac{{i{w^*}{e^{i{w^*}}}\delta }}{{{\epsilon ^*}({e^{i{w^*}}} + {\epsilon^*}{{\bar d}^T}N\theta )}}A\\ + \frac{{{\epsilon^*}}}{{({e^{i{w^*}}} + {\epsilon^*}{{\bar d}^T}N\theta )}}{{\bar d}^T}\left[ \begin{array}{l} f''({\xi ^*})({a_2}{{\bar \theta }_2} + {b_2}{\theta _2}) + \frac{1}{2}f'''({\xi ^*})\theta _2^2{{\bar \theta }_2}\\ g''({r^*})({a_1} + {b_1}) + \frac{1}{2}g'''({r^*}) \end{array} \right]{A^2}\bar A. \end{array} | (40) |
In this section, we consider the prototype system (4) and illustrate the results of section 3 by numerical simulation. In the following, a vector is said to be positive if all coordinates are positive. To have assumptions (B1) and (B2) hold for system (4), we show that,
Theorem 4.1. Consider system (4) with positive parameters
Proof. Since all parameters of system (4) are positive, direct computation shows that there exists a positive equilibrium. Then there exists a neighborhood
Consider the initial value problem associated with system (4). We obtain by the work of [23] that the initial value problem has a unique solution
Notice that if
Next we show that
\dot{\tau}(t) = \frac{\dot{x}(t)-\dot{x}(t-\tau(t))}{\frac{1}{c}-\dot{x}(t-\tau(t))} < 1. |
It follows that for every
\dot{x}(s) = -\mu_m x(s)+\frac{\alpha_m}{1+\left(\frac{y(s-\tau(s))}{\bar{y}}\right)^h} < \frac{\alpha_m}{1+\left(\frac{y(s-\tau(s))}{\bar{y}}\right)^h} < \alpha_m, |
which leads to
Next we show that
\dot{x}(t_0) =-\mu_m x(t_0)+\frac{\alpha_m}{1+\left(\frac{y(t_0-\tau(t_0))}{\bar{y}}\right)^h} = \frac{\alpha_m}{1+\left(\frac{y(t_0-\tau(t_0))}{\bar{y}}\right)^h}>0, |
which means that there exists
If
\dot{y}(t_0) =-\mu_p y(t_0)+\alpha_px(t_0-\tau(t_0)) = \alpha_px(t_0-\tau(t_0))>0, |
which means that there exists
\begin{align}\label{est-x} \dot{x}(t) =-\mu_m x(t)+\frac{\alpha_m}{1+\left(\frac{y(t-\tau(t))}{\bar{y}}\right)^h} < \frac{\alpha_m}{1+\left(\frac{y(t-\tau(t))}{\bar{y}}\right)^h} < \alpha_m. \end{align} | (41) |
It follows that
\begin{align}\label{est-y} \dot{y}(t) =-\mu_p y(t)+\alpha_px(t-\tau(t)) < \alpha_px(t-\tau(t)) < \alpha_p(\alpha_m t+x(0)). \end{align} | (42) |
By (41) and (42), and by the third equation of system (4),
Now we turn to assumption (B3). Let
\mu_m\mu_p = 1.2\times 10^{-3} < 5.9384374\times 10^{-3} =-f'(\xi^*)g'(r^*). |
Then by Lemma 3.1 and by Theorem 3.3, there exists a unique
\begin{align*} & (\lambda+\epsilon\mu_m)(\lambda+\epsilon\mu_p)-\epsilon^2f'(\xi^*)g'(r^*)e^{-2\lambda}\big|_{\, \lambda = 2iw^*, \, \epsilon = \epsilon_0}\\ = &-0.9140361052+0.1856539388\, i\neq 0. \end{align*} |
That is, (B3) holds. By Theorem 3.4, the normal form of system (4) near
\begin{align*} A' = &(0.01841158248+0.04829902976\, i)\delta A \\ & + \left((2.114544332\, c^2+0.0008578251748\, c-0.001233336633)\right.\\ & \left.-i(1.469928514\, c^2+0.002534237744\, c-0.003599996653)\right)A^2\bar{A}, \end{align*} |
where the real part of the coefficient of the cubic term
2.114544332\, c^2+0.0008578251748\, c-0.001233336633, |
which is negative for
If
If
Starting from model (5) of differential equations with threshold type state-dependent delay, we obtained model (11) with constant delay equivalent to (5) under assumptions (B1), (B2) and (B3), by using a time domain transformation. System (11) provides a possibility to investigate the dynamics of the original system near the equilibrium points. We remark that the normal form (39) of the general type of system (11) obtained by using the method of multiple time scales is also applicable to the following type of system:
\begin{align}\label{tranformed-general-c-1} \left\{ \begin{aligned} \frac{1}{ \epsilon} \frac{\mathrm{d}r}{\mathrm{d} \eta}& = {-\mu_m r(\eta)+f(\xi(\eta-1))}\\ \frac{1}{ \epsilon} \frac{\mathrm{d}\xi}{\mathrm{d} \eta}& = {-\mu_p \xi(\eta)+g(r(\eta-1))}, \end{aligned} \right. \end{align} | (43) |
which is equivalent to system (13). Even though many specific types of models of genetic regulatory dynamics have been developed in recent years (see [24], among many others), the general normal form we developed applies to a broad class of models.
Modeling the state-dependent delay of the genetic regulatory dynamics gives rise to the parameters
From the analysis of model (11), we find that the characteristic equation is independent of
A subcritical Hopf bifurcation usually means a sudden change of the dynamics when the system state is far enough from the stable equilibrium state. In terms of the model for the state-dependent delay
We remark that the numerical results in Section 4 is consistent with the observations from [2] described in the Section 1: when the analogous diffusion coefficient
We also remark that the attempt to model the state-dependent delay with an inhomogeneous linear function of the state variables reveals that the concentration gradients in the genetic regulatory dynamics may change the Hopf bifurcation direction modeled with constant delays. Since diffusion of substances in live cell may be confined or otherwise free and is a complex process [25], improved models for the state-dependent delay in the diffusion process may provide better means for understanding the underlying mechanisms of genetic regulatory dynamics. For instance, what if the stationary state of
The author would like to thank an anonymous referee for the detailed and constructive comments which greatly improved the paper.
[1] |
J. Yuh, Design and Control of Autonomous Underwater Robots: A Survey, Auton. Robot., 8 (2000), 7–24. doi: 10.1023/A:1008984701078. doi: 10.1023/A:1008984701078
![]() |
[2] |
K. H. Low, Maneuvering of biomimetic fish by integrating a bouyancy body with modular undulating fins, Int. J. Humanoid Robot., 4 (2007), 671–695. doi: 10.1142/S0219843607001217. doi: 10.1142/S0219843607001217
![]() |
[3] |
C. Ren, X. Zhi, Y. Pu, F. Zhang, A multi-scale UAV image matching method applied to large-scale landslide reconstruction, Math. Biosci. Eng., 18 (2021), 2274–2287. doi: 10.3934/MBE.2021115. doi: 10.3934/MBE.2021115
![]() |
[4] | C. I. Sprague, O. Ozkahraman, A. Munafo, R. Marlow, A. Phillips, P. Ogren, Improving the Modularity of AUV Control Systems using Behaviour Trees, AUV 2018 - 2018 IEEE/OES Auton. Underw. Veh. Work. Proc., Nov. 2018, doi: 10.1109/AUV.2018.8729810. |
[5] |
G. Ferri, A. Munafo, K. D. LePage, An Autonomous Underwater Vehicle Data-Driven Control Strategy for Target Tracking, IEEE J. Ocean. Eng., 43 (2018), 323–343. doi: 10.1109/JOE.2018.2797558. doi: 10.1109/JOE.2018.2797558
![]() |
[6] |
G. Salavasidis, A. Munafò, C. A. Harris, T. Prampart, R. Templeton, M. Smart, et al., Terrain-aided navigation for long-endurance and deep-rated autonomous underwater vehicles, J. F. Robot., 36 (2019), 447–474. doi: 10.1002/ROB.21832. doi: 10.1002/ROB.21832
![]() |
[7] |
W. Zhao, Y. Hu, L. Wang, Construction and Central Pattern Generator-Based Control of a Flipper-Actuated Turtle-Like Underwater Robot, Adv. Robot., 23 (2009), 19–43. doi: 10.1163/156855308X392663. doi: 10.1163/156855308X392663
![]() |
[8] | C. Zhou, K. H. Low, Kinematic modeling framework for biomimetic undulatory fin motion based on coupled nonlinear oscillators, in 2010 IEEE/RSJ Int. Conf. Intel. Robots Syst., 2010,934–939. doi: 10.1109/IROS.2010.5651162. |
[9] |
J. Yu, K. Wang, M. Tan, J. Zhang, Design and control of an embedded vision guided robotic fish with multiple control surfaces, Sci. World J., 2014 (2014), 631296. doi: 10.1155/2014/631296. doi: 10.1155/2014/631296
![]() |
[10] |
A. J. Ijspeert, A. Crespi, Online trajectory generation in an amphibious snake robot using a lamprey-like central pattern generator model, Proc. - IEEE Int. Conf. Robot. Autom., (2007), 262–268. doi: 10.1109/ROBOT.2007.363797. doi: 10.1109/ROBOT.2007.363797
![]() |
[11] |
D. Korkmaz, G. Ozmen Koca, G. Li, C. Bal, M. Ay, Z. H. Akpolat, Locomotion control of a biomimetic robotic fish based on closed loop sensory feedback CPG model, J. Mar. Eng. Technol., 20 (2021), 125–137. doi: 10.1080/20464177.2019.1638703. doi: 10.1080/20464177.2019.1638703
![]() |
[12] |
J.-K. Ryu, N. Chong, B.-J. You, H. Christensen, Locomotion of snake-like robots using adaptive neural oscillators, Intell. Serv. Robot., 3 (2009), 1–10. doi: 10.1007/s11370-009-0049-4. doi: 10.1007/s11370-009-0049-4
![]() |
[13] | M. Ikeda, K. Watanabe, I. Nagai, Propulsion movement control using CPG for a Manta robot, in The 6th Int. Conf. Soft Comput. Intel. Syst., and The 13th Int. Sympo. on Adv. Intel. Syst., 2012,755–758. doi: 10.1109/SCIS-ISIS.2012.6505174. |
[14] | L. Shang, S. Wang, M. Tan, Fuzzy Logic PID Based Control Design for a Biomimetic Underwater Vehicle with Two Undulating Long-fins, in India Conf. (INDICON) 2015 Annual IEEE, 2015, 1–6. |
[15] | J. Zhang, Multimodal swimming control of a robotic fish with pectoral fins using a CPG network, Chinese Sci. Bull., 57 (2012), 1209–1216. |
[16] | K. Inoue, S. Ma, C. Jin, Neural oscillator network-based controller for meandering locomotion of snake-like robots, in IEEE Int. Conf. Robot. Autom., 2004. Proc.. ICRA '04. 2004, 5 (2004), 5064–5069. doi: 10.1109/ROBOT.2004.1302520. |
[17] | C. Zhou, Modeling and control of swimming gaits for fish-like robots using coupled nonlinear oscillators, Nanyang Technological University, 2012. |
[18] |
V. D. Nguyen, D. K. Phan, C. A. T. Pham, D. H. Kim, V. T. Dinh, T. T. Nguyen, Study on Determining the Number of Fin-Rays of a Gymnotiform Undulating Fin Robot, Lect. Notes Electr. Eng., 465 (2018), 745–752. doi: 10.1007/978-3-319-69814-4_72. doi: 10.1007/978-3-319-69814-4_72
![]() |
[19] |
X. Dong, S. Wang, Z. Cao, M. Tan, CPG Based Motion Control for an Underwater Thruster with Undulating Long-Fin, IFAC Proc. Vol., 41 (2008), 5433–5438. doi: 10.3182/20080706-5-KR-1001.00916. doi: 10.3182/20080706-5-KR-1001.00916
![]() |
[20] |
A. Crespi, D. Lachat, A. Pasquier, A. J. Ijspeert, Controlling swimming and crawling in a fish robot using a central pattern generator, Auton. Robots, 25 (2008), 3–13. doi: 10.1007/s10514-007-9071-6. doi: 10.1007/s10514-007-9071-6
![]() |
[21] | M. Sfakiotakis, A. Manolis, N. Spyridakis, J. Fasoulas, M. Arapis, Development and Experimental Evaluation of an Undulatory Fin Prototype, in Proceedings of the RAAD 2013 22nd Int. Workshop on Robot. Alpe-Adria-Danube Region, 2013, no. May 2014, 1–8. |
[22] | M. Sfakiotakis, R. Gliva, M. Mountoufaris, Steering-plane motion control for an underwater robot with a pair of undulatory fin propulsors, in 2016 24th Mediterranean Conf. Control Autom. (MED), 2016,496–503, doi: 10.1109/MED.2016.7535989. |
[23] | V. H. Nguyen, V. D. Nguyen, V. T. Duong, H. H. Nguyen, T. T. Nguyen, Experimental Study on Kinematic Parameter and Undulating Pattern Influencing Thrust Performance of Biomimetic Underwater Undulating Driven Propulsor, Int. J. Mech. Mechatronics Eng., 20 (2020), 7. |
[24] |
W. Zhao, J. Yu, Y. Fang, L. Wang, Development of Multi-mode Biomimetic Robotic Fish Based on Central Pattern Generator, 2006 IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2006, doi: 10.1109/IROS.2006.281800. doi: 10.1109/IROS.2006.281800
![]() |
[25] |
X. Wu, S. Ma, CPG-based control of serpentine locomotion of a snake-like robot, Mechatronics, 20 (2010), 326–334. doi: 10.1016/j.mechatronics.2010.01.006. doi: 10.1016/j.mechatronics.2010.01.006
![]() |
[26] | R. Gliva, M. Mountoufaris, N. Spyridakis, M. Sfakiotakis, Development of a Bio-Inspired Underwater Robot Prototype with Undulatory Fin Propulsion, in 9th Int. Conf. on New Horiz. Ind. Bus. Edu. (NHIBE'15), 2015, 1–6. |
[27] |
Z. Lu, S. Ma, B. Li, Y. Wang, 3D Locomotion of a Snake-like Robot Controlled by Cyclic Inhibitory CPG Model, 2006 IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2006, doi: 10.1109/IROS.2006.281801. doi: 10.1109/IROS.2006.281801
![]() |
[28] | M. Wang, J. Yu, M. Tan, G. Zhang, A CPG-based sensory feedback control method for robotic fish locomotion, in Proceedings of the 30th Chinese Control Conf., 2011, 4115–4120. |
[29] |
C. Zhou, K. H. Low, On-line Optimization of Biomimetic Undulatory Swimming by an Experiment-based Approach, J. Bionic. Eng., 11 (2014), 213–225. doi: 10.1016/S1672-6529(14)60042-1. doi: 10.1016/S1672-6529(14)60042-1
![]() |
[30] |
M. Sfakiotakis, J. Fasoulas, R. Gliva, A. Yannakoudakis, Model-based fin ray joint tracking control for undulatory fin mechanisms, Int. Congr. Ultra Mod. Telecommun. Control Syst. Work., 2016 (2016), 158–165. doi: 10.1109/ICUMT.2015.7382421. doi: 10.1109/ICUMT.2015.7382421
![]() |
[31] |
C. Zhou and K. H. Low, Design and locomotion control of a biomimetic underwater vehicle with fin propulsion, IEEE/ASME Trans. Mechatronics, 17 (2012), 25–35. doi: 10.1109/TMECH.2011.2175004. doi: 10.1109/TMECH.2011.2175004
![]() |
[32] |
M. Sfakiotakis, J. Fasoulas, M. M. Kavoussanos, M. Arapis, Experimental investigation and propulsion control for a bio-inspired robotic undulatory fin, Robotica, 33 (2015), 1062–1084. doi: 10.1017/S0263574714002926. doi: 10.1017/S0263574714002926
![]() |
[33] |
P. M. Özturan, A. Bozanta, B. Basarir-Ozel, E. Akar, M. Coşkun, A roadmap for an integrated university information system based on connectivity issues: Case of Turkey, Int. J. Manag. Sci. Inf. Technol., 17 (2015), 1–23. doi: 10.14313/JAMRIS. doi: 10.14313/JAMRIS
![]() |
[34] |
K. H. Low, A. Willy, Biomimetic motion planning of an undulating robotic fish fin, JVC/Journal Vib. Control, 12 (2006), 1337–1359. doi: 10.1177/1077546306070597. doi: 10.1177/1077546306070597
![]() |
[35] |
R. Ruiz-Torres, O. M. Curet, G. V. Lauder, M. A. Maciver, Erratum: Kinematics of the ribbon fin in hovering and swimming of the electric ghost knifefish (Journal of Experimental Biology 216, (823-834)), J. Exp. Biol., 217 (2014), 3765–3766. doi: 10.1242/jeb.113670. doi: 10.1242/jeb.113670
![]() |
[36] |
K. H. Low, Modelling and parametric study of modular undulating fin rays for fish robots, Mech. Mach. Theory, 44 (2009), 615–632. doi: 10.1016/j.mechmachtheory.2008.11.009. doi: 10.1016/j.mechmachtheory.2008.11.009
![]() |
[37] |
I. English, H. Liu, O. M. Curet, Robotic device shows lack of momentum enhancement for gymnotiform swimmers, Bioinspir. Biomim., 14 (2019), 024001. doi: 10.1088/1748-3190/aaf983. doi: 10.1088/1748-3190/aaf983
![]() |
[38] |
I. D. Neveln, R. Bale, A. P. S. Bhalla, O. M. Curet, N. A. Patankar, M. A. MacIver, Undulating fins produce off-axis thrust and flow structures, J. Exp. Biol., 217 (2014), 201–213. doi: 10.1242/jeb.091520. doi: 10.1242/jeb.091520
![]() |
[39] | M. Ikeda, S. Hikasa, K. Watanabe, I. Nagai, A CPG design of considering the attitude for the propulsion control of a Manta robot, in IECON 2013 - 39th Ann. Conf. IEEE Ind. Electron. Soc., 2013, 6354–6358. doi: 10.1109/IECON.2013.6700181. |
[40] |
C. Liu, Q. Chen, D. Wang, CPG-inspired workspace trajectory generation and adaptive locomotion control for quadruped robots, IEEE Trans. Syst. man, Cybern. Part B, Cybern. a Publ. IEEE Syst. Man, Cybern. Soc., 41 (2011), 867–880. doi: 10.1109/TSMCB.2010.2097589. doi: 10.1109/TSMCB.2010.2097589
![]() |
[41] | C. M. A. Pinto, D. Rocha, C. P. Santos, Hexapod robots: New CPG model for generation of trajectories, J. Numer. Anal. Ind. Appl. Math., 7 (2012), 15–26. |
[42] |
T. Wang, W. Guo, M. Li, F. Zha, L. Sun, CPG Control for Biped Hopping Robot in Unpredictable Environment, J. Bionic Eng., 9 (2012), 29–38. doi: 10.1016/S1672-6529(11)60094-2. doi: 10.1016/S1672-6529(11)60094-2
![]() |
[43] |
S. Inagaki, H. Yuasa, T. Arai, CPG model for autonomous decentralized multi-legged robot system—generation and transition of oscillation patterns and dynamics of oscillators, Rob. Auton. Syst., 44 (2003), 171–179. doi: 10.1016/S0921-8890(03)00067-8. doi: 10.1016/S0921-8890(03)00067-8
![]() |
[44] |
M. Mokhtari, M. Taghizadeh, M. Mazare, Hybrid Adaptive Robust Control Based on CPG and ZMP for a Lower Limb Exoskeleton, Robotica, 39 (2021), 181–199. doi: 10.1017/S0263574720000260. doi: 10.1017/S0263574720000260
![]() |
[45] |
X. Wu, L. Teng, W. Chen, G. Ren, Y. Jin, H. Li, CPGs with continuous adjustment of phase difference for locomotion control, Int. J. Adv. Robot. Syst., 10 (2013), 1–13. doi: 10.5772/56490. doi: 10.5772/56490
![]() |
[46] |
Y. Cao, Y. Lu, Y. Cai, S. Bi, G. Pan, CPG-fuzzy-based control of a cownose-ray-like fish robot, Ind. Robot Int. J. Robot. Res. Appl., 46 (2019), 779–791. doi: 10.1108/IR-02-2019-0029. doi: 10.1108/IR-02-2019-0029
![]() |
[47] |
I. B. Jeong, C. S. Park, K. I. Na, S. Han, J. H. Kim, Particle swarm optimization-based central patter generator for robotic fish locomotion, 2011 IEEE Congr. Evol. Comput. CEC 2011, (2011), 152–157, doi: 10.1109/CEC.2011.5949612. doi: 10.1109/CEC.2011.5949612
![]() |
[48] | M. C. Chen Wang, G. Xie, L. Wang, CPG-based locomotion control of a robotic fish: Using linear oscillators and reducing control parameters via PSO, Int. J. Innov. Comput. Inf. Control, 7 (2011), 4237–4249. |
[49] |
J. Yu, Z. Wu, M. Wang, M. Tan, CPG Network Optimization for a Biomimetic Robotic Fish via PSO, IEEE Trans. Neural Networks Learn. Syst., 27 (2016), 1962–1968. doi: 10.1109/TNNLS.2015.2459913. doi: 10.1109/TNNLS.2015.2459913
![]() |
[50] |
J. Lee, S. Lee, S. Chang, B.-H. Ahn, A Comparison of GA and PSO for Excess Return Evaluation in Stock Markets, Lect. Notes Comput. Sci., 3562 (2005), 221–230. doi: 10.1007/11499305_23. doi: 10.1007/11499305_23
![]() |
[51] | C. Niehaus, T. Röfer, T. Laue, Gait Optimization on a Humanoid Robot using Particle Swarm Optimization, 2007. |
[52] |
Y. Zou, T. Liu, D. Liu, F. Sun, Reinforcement learning-based real-time energy management for a hybrid tracked vehicle, Appl. Energy, 171 (2016), 372–382. doi: 10.1016/j.apenergy.2016.03.082. doi: 10.1016/j.apenergy.2016.03.082
![]() |
[53] |
T. Liu, Y. Zou, D. Liu, F. Sun, Reinforcement learning-based energy management strategy for a hybrid electric tracked vehicle, Energies, 8 (2015), 7243–7260. doi: 10.3390/en8077243. doi: 10.3390/en8077243
![]() |
[54] |
R. C. Hsu, C. T. Liu, D. Y. Chan, A reinforcement-learning-based assisted power management with QoR provisioning for human-electric hybrid bicycle, IEEE Trans. Ind. Electron., 59 (2012), 3350–3359. doi: 10.1109/TIE.2011.2141092. doi: 10.1109/TIE.2011.2141092
![]() |
[55] |
H. Lee, C. Kang, Y. Il Park, N. Kim, S. W. Cha, Online data-driven energy management of a hybrid electric vehicle using model-based Q-learning, IEEE Access, 8 (2020), 84444–84454. doi: 10.1109/ACCESS.2020.2992062. doi: 10.1109/ACCESS.2020.2992062
![]() |
[56] |
T. Liu, X. H, S. E. Li, D. Cao, Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle, IEEE/ASME Trans. Mechatronics, 22 (2017), 1497–1507. doi: 10.1109/TMECH.2017.2707338. doi: 10.1109/TMECH.2017.2707338
![]() |
[57] |
Y. Lu, R. He, X. Chen, B. Lin, C. Yu, Energy-efficient depth-based opportunistic routing with q-learning for underwater wireless sensor networks, Sensors (Switzerland), 20 (2020), 1–25. doi: 10.3390/s20041025. doi: 10.3390/s20041025
![]() |
[58] |
R. Plate, C. Wakayama, Utilizing kinematics and selective sweeping in reinforcement learning-based routing algorithms for underwater networks, Ad Hoc Networks, 34 (2015), 105–120. doi: 10.1016/j.adhoc.2014.09.012. doi: 10.1016/j.adhoc.2014.09.012
![]() |
[59] |
Y. He, L. Xing, Y. Chen, W. Pedrycz, L. Wang, G. Wu, A Generic Markov Decision Process Model and Reinforcement Learning Method for Scheduling Agile Earth Observation Satellites, IEEE Trans. Syst. Man. Cybern. Syst., 1–12, 2020. doi: 10.1109/tsmc.2020.3020732. doi: 10.1109/tsmc.2020.3020732
![]() |
[60] |
Z. Jin, Y. Ma, Y. Su, S. Li, X. Fu, A Q-learning-based delay-aware routing algorithm to extend the lifetime of underwater sensor networks, Sensors (Switzerland), 17 (2017), 1–15. doi: 10.3390/s17071660. doi: 10.3390/s17071660
![]() |
[61] |
D. Zhang, Z. H. Ye, P. C. Chen, Q. G. Wang, Intelligent event-based output feedback control with Q-learning for unmanned marine vehicle systems, Control Eng. Pract., 105 (2020), 104616. doi: 10.1016/j.conengprac.2020.104616. doi: 10.1016/j.conengprac.2020.104616
![]() |
[62] |
Z. Chen, B. Qin, M. Sun, Q. Sun, Q-Learning-based parameters adaptive algorithm for active disturbance rejection control and its application to ship course control, Neurocomputing, 408 (2020), 51–63. doi: 10.1016/j.neucom.2019.10.060. doi: 10.1016/j.neucom.2019.10.060
![]() |
[63] |
Y. Nakamura, T. Mori, S. Ishii, Natural Policy Gradient Reinforcement Learning for a CPG Control of a Biped Robot, Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 3242 (2004), 972–981. doi: 10.1007/978-3-540-30217-9_98. doi: 10.1007/978-3-540-30217-9_98
![]() |
[64] | T. Mori, Y. Nakamura, M. A. Sato, S. Ishii, Reinforcement learning for a CPG-driven biped robot, Proc. Natl. Conf. Artif. Intell., (2004), 623–630. |
1. | Michael Eden, Adrian Muntean, Homogenization of a fully coupled thermoelasticity problem for a highly heterogeneous medium witha prioriknown phase transformations, 2017, 40, 01704214, 3955, 10.1002/mma.4276 | |
2. | Michael Eden, Hari Shankar Mahato, Homogenization of a poroelasticity model for fiber‐reinforced hydrogels, 2022, 45, 0170-4214, 11562, 10.1002/mma.8466 | |
3. | Michael Eden, Tom Freudenberg, Effective Heat Transfer Between a Porous Medium and a Fluid Layer: Homogenization and Simulation, 2024, 22, 1540-3459, 752, 10.1137/22M1541794 |