Online intelligent maneuvering penetration methods of missile with respect to unknown intercepting strategies based on reinforcement learning

1.   Introduction

The study of maneuvering penetration technologies which can improve the combat effectiveness of missiles is a hot topic in the research field of guidance and control, there are many methods about midpiece maneuvering penetrations, such as pre-procedural maneuvering penetration strategies ^[1]; actively evasive maneuvering penetration methods ^[2]; penetration methods based on flying around the detection interception area ^[3] and so on.

In the process of pre-procedural maneuvering penetration methods, the trajectories of penetration are preset before launching, the missiles can not make any responses with respect to strategies of the interceptor, alternatively, the missile flies along the preconcerted trajectories and is abided by the preconcerted maneuvering timing. Compared with the pre-procedural maneuvering penetration strategies, the actively evasive maneuvering penetration strategies, which can obtain the the optimal penetration strategies based on the parameters of interceptor detected by missile-borne computer, can increase the rate of successful penetration. There are two mainly maneuvering guidance strategies in the actively evasive maneuver penetrations, one is differential game type ^[3], the other is matrix game type ^[4]. From the mathematical standpoint, to derive the differential game typical guidance strategies is equivalent to solve a bilateral extremum problems for the associated functionals, but it is very hard to find the analytical solutions for these kinds of functionals. With respect to achieve the numerical solutions, it costs a lot of computing resource of the missile-borne computer to keep the high precision and real-time performance. In light of the finite two-person zero-sum game theory, matrix game typical guidance strategies consider the missile and interceptor as the players, the target-missing quality and its negative value are regraded as their payments. This method needs a lot of information on the motion of the interceptor, for more details, one can refer to ^[5].

Recently, with the improvement of the computing ability of computer, the artificial intelligence have been developed rapidly ^[6], investigations on theory and applications of reinforcement learning (RL)^{[7,8,9,10,11,12,13]} is very important. In order to obtain the best operating action of the whole system, the RL can make the intelligent agent to select the behaviors which can gain the maximum reward of the environment state by learning the mappings from the environmental states to the behaviors. There are many literatures on the theory and application of RL. For instance, Bradtke and Duff ^[14] considered the continuous-time Markov decision problems by using RL. Taking advantage of the RL, some problems in transport were studied by Abdulhai and Kattan ^[15]. Based on the RL, Lewis et al. ^[16] designed optimal adaptive controllers by using natural decision methods. In the achievements on dynamics and control problem of robotics, the RL was also employed wildly ^[17]. As for the applications of RL in economics, one can refer to ^[18]. With respect to literatures on dynamics and control of missiles or aircrafts by RL, Shalumov ^[19] proposed cooperative online guide-launch-guide policy for the target-missile-defender engagement. The computational guidance problem of missile was considered in ^[20]. A homing-phase guidance law of missile was considered in ^[21]. The scenarios of avoiding Obstacles via missile real-time inference belong to Hong and Park ^[22]. Gaudet et al. gave an angle-only intercept guidance of maneuvering targets ^[23]. missile guidance for head-on interception of maneuvering target was established by Li et al. ^[24]. A planar evasive maneuvering strategy of aircrafts was derived in ^[24], too name but a few. For more detail, one can refer to ^{[11,25,26,27,28]}.

Actually, RL is a kind of self-regulated learning method driven by experiences, the maneuvering penetration technologies based on RL are nearly trained with predictable intercepting strategies, which means that the maneuvering penetration technologies are useless if the intercepting strategies can not be acquired beforehand. In order to overcome this obstacle, this paper is devoted to propose online intelligent maneuvering penetration methods with respect to non-predictable intercepting strategies based on RL, the main idea is that: let the tentative maneuvers of missile and line of sight rates (LOS) rate between missile and interceptor be actions and states respectively. Inducing the interceptor to generate the responses with respect to the actions, which can be captured by the missile-borne detectors. In the light of the increment of LOS rates, the reward function can be designed. Thus, by means of the information gathered by missile-borne detectors, the the maneuvering penetration strategies can be derived by missile-borne computer based on RL.

The rest of this paper is organised as follows. The preliminaries are given in Section 2. Section 3 is devoted to demonstrate the main results, in which the online intelligent maneuvering penetration methods are proposed and some numerical results are accomplished to validate this new method. In Section 4, the conclusions are listed.

2.   Preliminaries

For brevity, some symbols are introduced firstly, let $D$ and $M$ be the interceptor and the missile respectively, ${\bf r} = (r_x, r_y, r_x)$ is the relative distance between the missile and the interceptor. Based on the kinematic theory, the relationship between the missile and the interceptor in attack-defense confrontation can be described by Figure 1. $\theta_1$ is the ballistic inclination angle of the interceptor, the ballistic declination angle of the interceptor is denoted by $\varphi_1$. $\theta_2, \varphi_2$ are the ballistic inclination angle and the ballistic declination angle of missile respectively. $v_{M}$ is the velocity of missile, and the velocity of interceptor is signified by $v_D$, thus $v_r = v_D-v_M$ is the relative velocity between the missile and the interceptor. ${\bf q} = (q_{x}, q_{y}, q_{z})$ is LOS angle in the ground coordinate system, the horizontal and vertical LOS angle are denoted by $q_{\varphi}$ and $q_{\theta}$. Let $(x_{M}, y_{M}, z_{M})$ and $(x_{D}, y_{D}, z_{D})$ be the positions of missile and interceptor in ground coordinate system respectively.

By ^[29], the mathematical model of relative motion between the missile and the interceptor can be described as follows

the equation governing the dynamics of LOS angle is

the position of missile can be described by

the following equations model the position of the interceptor

Next, it is turned to give the description on interceptor guidance blind area.

In general, exoatmospheric kill vehicle (EKV), which consists of guidance systems, transferring orbital control systems, propulsion systems and so on (for more detail, see Figure 2), destroys the missile by collision. The homing guidance system should be interrupted when the distance between EKV and missile is less than or equal to a certain quantity $R$ which is referred as guidance blind area. Obviously, the EKV is uncontrollable in guidance blind area, in addition, the interceptor guidance blind area is about 30 m to 500 m in practice. Figure 3 indicates the graphical representation on instantaneous miss distance $d$ of EKV ^[30,31] which can be determined as following,

where $R$ is the instantaneous distance between the missile and the interceptor, $\dot{R}$ denotes the instantaneous relative velocity when both the missile and the interceptor just enter the guidance blind area, $\|\dot{{\bf q}}\| = \sqrt{\dot{q}_{\varphi}^2+\dot{q}_{\theta}^2}$. Clearly, the instantaneous miss distance is proportional to $\|\dot{{\bf q}}\|, \; R$ and is inversely proportional to $|\dot{R}|$. Since the time of both missile and EKV are in guidance blind area is so short that the missile can not maneuver, thus, the instantaneous miss distance can be regarded as the actual miss distance.

The RL can be divided into two kinds, one is using the same strategy to update the value function and select a new action, such as Sarsa method. The other is utilizing the different strategies to do these things, for instance, Q-learning (QL) method. In QL, the interactions between agent and environment are modelled by the markov decision process $(\mathcal{S}, \mathscr{A}, \mathbb{P}, \overline{R}, \gamma)$, where $\mathcal{S}$ is the states, $\mathscr{A}$ is a $\sigma-$ algebra generated by the sets which is composed of all possible actions, $\mathbb{P}$ is the transition probability of one state to another state, $\overline{R}$ is the reward function, $\gamma$ is discount factor. Let $S_t\in\mathcal{S}$ be the state at time $t$, $A_t\in\mathscr{A}(S_t)$ is the action, here $\mathscr{A}(S_t)$ is the set of actions available in state $S_t$, for simplicity, we use $\mathscr{A}(t)$ instead of $\mathscr{A}(S_t)$. $\overline{R}_t\in\overline{R}$ is the reward, and $S_{t+1}$ is the next state of the agent (see Figure 4). Let

where $0\leq \gamma\leq 1$ is the discount rate. For all $s', s\in\mathcal{S}$, $r\in\overline{R}$ and $a\in\mathscr{A}(s)$, let

suppose $\pi$ is the policy, $\pi_{*}$ is the optimal policy, $\pi(a|s)$ is the probability of getting the action $a$ at time $s$, then the associated state-value function is given as following

the action-value function is

Together with (2.6)–(2.9) and the markov property, we have

and

thus, the optimal state-value function is

and the optimal action-value function is

Generally, the following $\epsilon-greedy$ strategy is employed in QL

where $Q(s, a)$ is the estimated value of the action value function ${\overline{q}}_{\pi_*}(s, a))$.

The flow chart of QL algorithm is as following, where $\mathcal{S^+}$ is the state space containing the termination state, and

3.   Main results

3.1.   Design on the intelligent maneuvering penetration strategy

Firstly, we propose the flow chart of the maneuvering penetration methods of the missile with respect to unknown intercepting strategies by QL methods (see Figure 5).

Dividing the combat process between the missile and the interceptor into three stages. After the missile is locked by the interceptor, let the missile make tentative maneuvers to induce the interceptor to correspondingly generate the responses, based on the information of responses of the interceptor, maneuvering penetration strategies of the missile are trained by QL method. This is the training stage. After that, the missile do not make any tentative maneuver, thus, the second stage is named as level flight stage. The third stage is referred to as penetration stage, the missile produces the maneuvering penetration strategies in terms of the training results and calculate instantaneous miss distance of EKV in guidance blind area to implement successful penetration.

It is emphasized the maneuvering penetration strategies indicated above do not need to know the law of the interceptor beforehand, alternatively, the necessary information for QL training is acquired by missile-borne detectors, and the maneuvering penetration strategies based on QL training is done by the missile-borne computer. Thus, the maneuvering penetration strategies are intelligent and real-time.

In the following, we realize the flow chart described by Figure 5. To begin with, the following markov decision process model is established.

$1)$ The states space: the instantaneous antagonistic states between the missile and the interceptor are depended on the LOS rates, thus, let the LOS rates be the state with the following form

the unit is $rad/s$.

$2)$ The actions space: the offense-defense confrontation between the missile and the interceptor can be formulated by pursuit-evasion game, the actions of which consist of up, down, left and right. In order to consider the penetration ability of missile with lower maneuverability, we set the optional overloads are $2$ and $3.5$ along with each direction of the up, down, left and right, the ultimate overload of interceptor is supposed to be $8$, thus, we establish the actions of missile as follows

$3)$ The reward function: the purpose of the test maneuvers of missile is to find the sequences which can result in the increasing of LOS rates. In order to generate a large value of LOS rate during the penetration, which can lead to the failure of proportional guidance method of EKV, the value of reward should be big when LOS rate is big and vice versa. Together with (2.5) and the amount of instantaneous off-target required, the critical value of LOS rate can be derived. If the LOS rate is larger than this critical value, the bigger value of reward function should be given. In practice, the LOS rates is very small (its magnitude is less than $10^{-1}$), thus, the rewards are set to be the absolute value of the LOS rates when the LOS rates are less than the critical value. In other cases, let the value of reward be 10. As indicated above, the reward function is designed as the following form

$4)$ The behavioral strategies: following the $\varepsilon-$ greedy strategy, the behavioral strategies of missile can be set up.

3.2.   Numerical results

This subsection is devoted to list the numerical results to validate the intelligent maneuvering penetration strategies proposed. Let the initial velocity of both the missile and the interceptor are $900\; m/s$, the initial positions are $(120,000\; m, -120,000\; m, 0\; m)$ and $(0\; m, 0\; m, 120,000\; m)$ respectively. Suppose strategy of the interceptor is proportional navigation method ^[32], the coefficient of which is $Ne = 3$, it is mentioned that the missile is not trained with respect to this strategy before launching. We can get tangential acceleration of interceptor by proportional navigation method, and the tangential acceleration of missile can be derived from the actions in QL. Set the learning rate $\alpha = 0.5$, discount rate $\gamma = 0.9$ in QL, the initial of $\varepsilon$ in $\varepsilon-$greedy strategy is $0.6$, which is decreased with the increasing of the frequency of training.

From the missile is locked by interceptor to the relative distance is $100,000\; m$ is the training stage, the relative distance between $25,000\; m$ to $100,000\; m$ is the level flight stage, the penetration stage is ended at the missile enter to the guidance blind area and calculate out the instantaneous miss distance or the relative distance is less than or equal to $300\; m$. In order to make comparison, the no maneuvering strategies and random maneuvering strategies with the same initial condition for missile are also considered. With these two strategies, dividing the combat process between missile and interceptor into two stages: from the missile is locked by interceptor to the relative distance is $25,000\; m$ is the level flight stage, after that, the missile enter the penetration stage.

From Table 1, we can assert that, under the QL maneuver, the missile produces large LOS rates to induce the overload of the interceptor to exceed the ultimate value, which gives rise to that the miss distance is $64.04\; m$. The Figure 6 illustrates the flight trajectory of the missile and the interceptor EKV with QL maneuver. Figures 7 and 8 show the variations of the overloads for the missile and the interceptor. Figure 7(a) interprets that, in the case of no maneuver, the missile can be effectively intercepted by switching the overload of EKV smoothly. In the circumstance of random maneuver, although the maneuvers produced by missile induce the associated responses of the interceptor, the overload of the interceptor do not over the ultimate value, which indicates that the interceptor have enough residual capacity to intercept the missile and the instantaneous miss distance is small, see Figure 7(b). By the lateral and longitudinal overload in the case of QL maneuver painted in Figure 8, the overload of EKV can exceed the ultimate value in the case of QL maneuver and the instantaneous miss distance is large, which means that the interceptor can not intercept missile.

The changes of LOS rates are described by Figures 9 and 10. With the QL maneuvering strategies, we see that the LOS rates increase dramatically in the penetration stage, increasing the overload of interceptor to the ultimate value can not restrain the divergence of the LOS rates, which results in the off-target of the interceptor, see Figure 10. From Figure 9, we find that the LOS rates are almost unchanged, thus, the miss distance is small.

In order to formulate the probability of successful penetration with the three maneuvering strategies, let $d_1 = 6.5$, here $d_1$ is the critical value of miss distance, if the miss distance $d_0 > d_1$, it accounts for the penetration is successful, otherwise, the penetration is failed. With the same initial condition, take $1000$ numerical results as a group, $10$ groups of numerical simulation are accomplished for each strategy. The numerical results state that the probability of successful penetration with no maneuvering strategies or random maneuvering strategies is $0$, the probability of successful penetration with QL maneuvering strategies is about $80\%$ (see Table 2).

From Figure 11, we obtain the mean of miss distances with random maneuver is about $0.2\; m$, the variance is small, it is the reason that the probability of successful penetration is nearly $0$ when we set the critical value of miss distance is $6.5\; m$. When the missile with the QL maneuvering strategy, the mean of miss distance is about $60\; m$, although the variance of miss distance is larger than in random maneuver, combined with Table 2, we can assert that the QL maneuvering strategies can produce effective penetration with high probability. Furthermore, the Figure 12 describes that the miss distances are distributed in $[0,200\; m]$ and concentrated around $50\; m$, thus, we can guarantee that the QL maneuvering strategy can leads the missile to realize successful penetration steady.

The Figure 13 illustrates the target chart of miss distances in the cases of random maneuvering strategies and QL maneuvering strategies. In the case of random maneuver, the distribution of target points are random, no obvious regularity can be found intuitively, see Figure 13(a). Because of the limit on the times of training, the most of target points are around one of the coordinate axis under the QL maneuvering strategies.

In order to illustrate the intensively effectiveness of maneuvering strategies proposed, we set proportional guidance coefficients $Ne$ of EKV to be $2.3$, $2.5$, $2.7$ and $3.3$ respectively, under which the similar numerical investigations are considered. The related results are given in Table 3.

As indicated above, we can conclude that, in the confrontation between the missile and the interceptor, the missile trained by QL maneuver based on the information of tentative maneuver of missile and the responses of interceptor induced can autonomously select a suitable maneuvering strategy to implement penetration successfully.

4.   Conclusions

When the intercepting manners are unknown in the confrontation between the missile and the interceptor, this paper proposes the intelligent maneuvering penetration strategies of the missile based on reinforcement learning, which is referred as QL maneuver. Compared with the no maneuvering and random maneuvering methods, the QL maneuver possesses the advantage of high probability of successful penetration. To be important, it is only needed the information of positions and LOS rates in the process of training the QL maneuvering strategies, furthermore, the time of training is not very long and all the training is done by the missile-borne computer when the missile are flying.

Acknowledgments

This work have been supported by National Natural Science Foundation of China (No. 12072178; No. 12002194; N0. 12202250), Project (No. ZR2020MA054; No. ZR2020QA037) supported by Shandong Provincial Natural Science Foundation.

Conflict of interest

The authors declare there is no conflicts of interest.