Research on tracking strategy of manipulator based on fusion reward mechanism

1.   Introduction

The manipulator is a widely used device in aviation, industry, medicine, and other fields, replacing human beings to complete high-precision, high-intensity repetitive tasks and greatly improving social productivity. Although traditional motion control strategies ^[1] can achieve high accuracy, they are only applicable to the operation of target objects under specific conditions. When the task or execution environment changes, it is necessary to re-extract motion features and design the motion attitude or re-code a new task for implementation, which limits intelligence, generalization, and productivity development. To enable a manipulator to replace human beings in a complex and changeable unstructured environment ^[2], it must have enough intelligence and generalization ability to deal with complex and changeable environments. Future environments for manipulators to perform tasks will become more and more complex and unstructured. How to make a manipulator calmly cope with this complex environment and successfully perform tasks has become the focus of artificial intelligence research.

The continuous promotion of Industry 4.0 ^[3] has enabled humanity to enter the fourth industrial revolution, dominated by intelligent manufacturing, where reinforcement learning algorithms play a pivotal role. In the field of intelligent manufacturing ^[4], the emergence of reinforcement learning algorithms has provided a new solution for the manipulator to work in high-dimensional and complex environments. Although early reinforcement learning algorithms such as Q-learning ^[5] enable the manipulator the intelligence of active exploration and updating strategies to a certain extent, they rely too much on Q tables, and the time and space costs of searching and storage are too high; also, they cannot be applied to solve high-dimensional problems.

With the rapid development of deep learning, reinforcement learning researchers have applied neural network principles to reinforcement learning technology, bringing a new round of development opportunities for deep reinforcement learning ^[6] (e.g., DQN ^[7], Dueling DQN ^[8], Double DQN ^[9], Rainbow DQN ^[10]). While the DQN series algorithms have successfully addressed the time and space complexity issues associated with the Q table, having achieved notable progress in solving high-dimensional problems, their applicability is more suited to discrete motion spaces. Unfortunately, they fall short when it comes to addressing continuous problems, such as the movement environment of a spatial manipulator. Schulman et al. ^[11] put forward the trust region policy optimization algorithm (TRPO), which is very effective for optimizing large nonlinear strategies ^[12] and solving high-dimensional continuous problems. However, the constraint of the TRPO algorithm is that there is little difference between the old and new strategies, and it has the disadvantage of consuming too many computing resources. Schulman et al. ^[13] proposed a PPO algorithm to directly constrain the probability ratio of new and old strategies so that the computational resource consumption is reduced, and the overall performance of the algorithm is better than TRPO. For solving high-dimensional continuous problems, the balance performance of the PPO algorithm is better than other deep reinforcement learning algorithms. Therefore, this paper implements the tracking control task ^[14] of the manipulator based on the PPO algorithm.

Regarding the manipulator tracking control application scenario, to ensure it has enough intelligence and generalization ability to cope with complex and changing real-world environments, this paper designs a PPO algorithm based on a convolutional neural network (CNN) that incorporates a reward improvement mechanism, including trajectory correction reward, core area acceleration guidance reward, ladder adaptability reward, and abnormal termination penalty. On one hand, the issue of reward sparsity in tracking control tasks for the manipulator is addressed; on the other hand, it also solves to some extent the problem of a lack of exploration and blind ineffective exploration of the PPO algorithm itself. Finally, the results of simulation experiments in different ranges show that the mean value of tracking success rate of the manipulator under a fused reward mechanism is higher than that of a sparse reward mechanism by 5 to 7 percentage points, and the mean value of the number of steps required for successful tracking with the fused reward mechanism is around 9.7, lower than that of the sparse reward mechanism by about 13. Thus, the manipulator of the PPO algorithm with the improved fused reward mechanism has higher accuracy and faster execution speed in tracking and control tasks.

The remaining paper is organized as follows: Section 2 introduces the basic principle of PPO algorithm. Section 3 displays the design of the manipulator tracking strategy based on the fusion reward mechanism. Section 4 presents the experimental results and analysis. Section 5 gives the conclusion.

1.1.   Related work

The primary challenge in deep reinforcement learning lies in the issue of sparse rewards, which leads the manipulator to engage in blind random exploration ^[15]. Experts have proposed various solutions from different perspectives. Vecerik et al. ^[16] introduced a demonstration method that partially addressed the sparse reward problem. Colas et al. ^[17] proposed the GEP-PG algorithm, aiming to resolve the sparse reward problem by optimizing the performance of the exploration phase. Conti et al. ^[18] combined the novelty search algorithm (NS) ^[19] and the quality diversity (QD) ^[20] algorithm with evolutionary strategies, creating a new algorithm that effectively improved performance in a sparse reward environment. Riedmiller et al. ^[21] presented scheduling-assisted control (SAC-X), capable of learning complex behaviors from scratch in the presence of multiple sparse reward signals. Wang et al. ^[22] suggested a non-expert-assisted deep reinforcement learning (RL) algorithm, enhancing its suitability for addressing sparse reward problems. Qi et al. ^[23] proposed a novel hybrid-vehicle energy management strategy algorithm based on reinforcement learning, effectively addressing the sparse reward problem during vehicle power allocation task training. Li et al. ^[24] introduced a stochastic curiosity-driven model rooted in deep reinforcement learning to address reward sparsity through internal reward guidance. Liu et al. ^[25] proposed cooperative multi-agent exploration (CMAE), fostering exploration through cooperation between multi-agents to overcome the impact of sparse rewards. Xia et al. ^[26] introduced the SparKGR hybrid multi-hop reasoning model, optimizing the guidance strategy to improve the reasoning performance of sparse knowledge graphs. Cai et al. ^[27] suggested a reward shaping and discounting scheme based on LDGBA that relies solely on the EP-MDP state, overcoming the problem of reward sparseness. Roghair et al. ^[28] proposed a convergence-based method and a domain network-based guidance approach to address the sparse reward problem in UAV flight training. Eoh et al. ^[29] introduced an automatic process learning (ACL) method for object transfer based on deep reinforcement learning (DRL) to address the sparse reward problem. Christianos et al. ^[30] proposed the SEAC algorithm, facilitating experience sharing between agents for effective exploration and partially resolving the issue of reward sparsity. Liu et al. ^[36] proposed a deep reinforcement learning-based autonomous obstacle avoidance combined with SLAM for a wheeled snake robot, by constructing a 3D model of the snake robot in Gazebo, which is able to achieve effective path planning and environment mapping in an environment with obstacles. Wen et al. ^[37] proposed the maximum entropy algorithm LGE-SAC based on prioritized experience replay, which effectively avoids obstacles and successfully reaches the goal through simulation experiments in a Gazebo simulator environment and real experiments on a Turtlebot3 robot equipped with a LiDAR sensor. Azar et al. ^[38] used a deep reinforcement learning (DRL) approach to robot navigation and exploration, integrating learned strategies and waypoint selection to develop a complete autonomous navigation and exploration system. Haider et al. ^[39] introduced the most important DRL-based navigation and control algorithms for mobile robots, describing the subcomponents of navigation perception, mapping, localization, and motion planning for mobile robots. Ahmed et al. ^[40] proposed an adaptive fixed-time fractional-order integral control for externally perturbed Euler-Lagrange systems, which has better tracking and convergence performance when compared to adaptive fractional-order sliding mode control schemes. Jahanshahi et al. ^[41] provided a comprehensive overview of the integration of machine learning techniques into robotic grasping, with a particular emphasis on the challenges and advances in spatial applications, providing valuable insights into the integration of deep reinforcement learning in robotic manipulation tasks. The aforementioned studies aim to overcome the sparse reward problem from several perspectives and approaches and demonstrate unique advantages and technical features in their specific application scenarios. However, when facing complex unstructured application scenarios, these approaches may exhibit certain limitations in terms of accuracy and generalization ability.

1.2.   Motivation and contribution

The algorithms above provide solutions to the sparse reward problem in a variety of application fields but lack pertinence and applicability in the tracking control task of manipulators.

To enhance the tracking and guidance capabilities of the manipulator in a multi-dimensional space and reduce its blind randomness in the detection and sampling process, this study proposes an enhanced proximal policy optimization (PPO) reinforcement learning algorithm. The improvement is achieved by incorporating a convolutional neural network (CNN) for image feature extraction and introducing a fusion reward mechanism. This paper provides a detailed exposition of the CNN model and outlines the fusion reward mechanism. Experimental trials are conducted to validate the proposed algorithm, and corresponding results are presented for comparative analysis.

The primary innovation of this study is the implementation of an enhanced PPO algorithm that integrates the manipulator and the environment. The algorithm incorporates a synthesis of reward mechanisms, encompassing trajectory correction rewards, core area adaptive rewards, and stage adaptive rewards and punishments. These components collaborate to facilitate the manipulator's rapid and seamless learning of the target object's maneuvering strategy, thereby enhancing the overall efficiency of the algorithm.

2.   Deep reinforcement learning algorithm

2.1.   Principle of deep reinforcement learning algorithm

Reinforcement learning algorithms are essentially end-to-end intelligent control decision-making algorithms. Deep reinforcement learning algorithms integrate deep learning with reinforcement learning, mainly including five elements: environment, agent, action, reward, and state.

Figure 1 shows the basic interaction process between the agent and the environment. The agent first obtains the state and reward from the environment, then outputs the action, and acts on the environment by updating the reinforcement learning algorithm. Finally, the agent continues to interact with the environment, iteratively updating state, reward, and action through the current state and current reward, obtaining as many cumulative rewards as possible, and ultimately obtaining the action strategy that best suits the current environment (that is, the strategy that can satisfy the maximum reward). Deep reinforcement learning combines the powerful feature extraction ^[31] capability of deep learning and the excellent decision-making capability of reinforcement learning.

2.2.   Fundamentals of the improved PPO algorithm

The PPO algorithm represents an improvement on the strategy gradient algorithm, which mainly solves the problems of low sampling efficiency and unstable training. The sampling method of the strategy gradient algorithm is based on Monte Carlo sampling ^[32]. In this method, the agent takes the current policy to sample the current environment for an episode and updates the policy according to the sampling data; the sampling data is used only once. This sampling method causes insufficient data utilization and low sampling efficiency, but the PPO algorithm uses importance sampling ^[33] to skillfully solve this problem. The following section delineates the two pivotal steps of the PPO algorithm: cumulative reward ^[34] and importance sampling.

(1) Cumulative rewards:

If the step of a sampling episode is defined as $\tau \left(\tau = {s}_{1}, {a}_{1}, {s}_{2}, {a}_{2}, {s}_{3}, {a}_{3}, \dots, {s}_{t}, {a}_{t}\right)$, the probability of $\tau$ can be expressed as:

Then, the cumulative rewards obtained in an episode are defined as:

A reward is the core of deep reinforcement learning algorithms and an important basis for guiding agents to make decisions. The difference in reward design greatly affects the result of the agent's task execution. Sparse rewards will make the agent's convergence speed very slow or even nonconvergent. Therefore, a fusion reward mechanism is designed to solve this problem.

(2) Importance sampling:

Importance sampling is essentially an approximate sampling method. Supposing there are two distributions $p\left(x\right)$ and $q\left(x\right)$, and knowing that $p\left(x\right)$ satisfies a certain distribution but $p\left(x\right)$ cannot be integrated and can only be sampled from $q\left(x\right)$, then for a distribution function $f\left(x\right)$ whose independent variable x satisfies $p\left(x\right)$, its expectation is:

The policy update gradient of the PG algorithm is as follows:

Transform Equation 2.3 to get:

This makes the algorithm perform the transition from on-policy to off-policy. This way, the collected data can be updated several times, reusing data and greatly improving data utilization.

3.   Design of manipulator tracking strategy based on a fusion reward mechanism

3.1.   Setting neural network parameters

In network structure configuration, the actor network and the critic network of the PPO algorithm exhibit congruence in their internal composition. The network comprises four convolutional layers and two fully connected layers. Each convolutional layer is equipped with a 3 × 3 convolutional kernel, with a step size of 2, a boundary padding of 1, and a nonlinear activation function using the ReLU function. Both the actor and the critic networks use images as the input, and the original image needs to go through a preprocessing step. Specifically, the RGB image is first converted to a grayscale image to reduce channel dimensions, the pixel value range is normalized from [0–255] to [0, 1], and the image size is cropped to 84 × 84 pixels; then, features are extracted from the image.

The primary distinction between the two networks lies in the input parameters; those of the actor network encompass the image and the end position determined by the camera, while the critic network receives the image captured by the camera. Subsequent to two fully connected layers, the actor network generates 3D action vectors, and the critic network yields scalars as a fitted estimate of the value function. The comprehensive structural design of the actor and critic networks is illustrated in Figure 2.

3.2.   Application of PPO algorithm in the manipulator tracking environment

The tracking application of the manipulator is a high-dimensional, complex, and continuous problem. The PPO algorithm provides an efficient and effective solution to continuous problems, as previously explained in the algorithm analysis. Consequently, the PPO algorithm is employed in the manipulator's tracking application.

Figure 3 illustrates the fundamental procedure of the PPO algorithm implementation within the manipulator tracking environment. Throughout the entire basic procedure, the target object emerges randomly within the white cube frame within the simulation environment, as illustrated in Figure 4. Subsequently, the tracking environment image data is acquired by the built-in camera of the PyBullet simulation. The environment state $s = \left[{x}_{e}, {y}_{e}, {z}_{e}, {x}_{b}, {y}_{b}, {z}_{b}\right]$($\left[{x}_{e}, {y}_{e}, {z}_{e}\right]$ is the coordinate position of the end-effector of the manipulator, and ${[x}_{b}, {y}_{b}, {z}_{b}]$is the coordinate of the target object. Data is transferred to the actor and critic networks of the CNN ^[35] architecture of the PPO algorithm through the Gym interface. The actor network outputs two parameter values, $\mu$ and $std$, according to the current state, uses these two values to sample actions from a Gaussian-distributed action policy, selects an action ${a}_{t}$ and passes it to the intelligent body, and then computes $a\_loss$ to backpropagate to the actor network to update the parameter ${\theta }^{\mu }$. The agent makes an action ${a}_{t}$, the environment state becomes ${s}_{t+1}$, and at the same time it gets a reward ${r}_{t}$. The state, action, and reward are stored as a tuple $\left({s}_{t}, {a}_{t}, {r}_{t}, {s}_{t+1}\right)$ in the experience pool. The critic network calculates the state value estimate $v$based on the current state ${s}_{t}$ and then back propagates to update the critic network parameters ${\theta }^{\mu }$. The range of motion of each joint is set to avoid exceeding the physical limits, and a smooth path planning algorithm is used to deal with the joint limits and singularities in the workspace to ensure the safety and efficiency of the manipulator's motion. This process iterates continuously to optimize the manipulator's path and control strategy.

3.3.   Fusion reward mechanism

The guidance mechanism composed of rewards is the core of the deep reinforcement learning algorithm to control the agent to achieve the goal task. The highly complex environment and wide exploration space of the manipulator make it difficult for the manipulator to track the target object with sparse reward guidance. When the manipulator successfully tracks the target, it obtains a reward of 1, and when it fails to track the target, it does not obtain a reward. This makes the reward very sparse, resulting in slow convergence or even inability to converge. To solve this problem, a fusion reward guidance mechanism was developed from multiple design perspectives, such as accelerated access to rewards in core areas and step-length guidance rewards.

Sparse reward is defined as follows:

Fusion reward is defines as follows:

(1) Trajectory correction reward

where ${r}_{1}$ is the trajectory correction reward, ${d}_{ao}$ is the distance from the end coordinate position of the manipulator to the target coordinate position at the current time calculated by Eq 3.2, and ${d}_{bo}$ is the distance from the end coordinate position of the manipulator at the previous time to the target coordinate position calculated by Eq 3.3. It is well-established that the sparseness of the reward signal can lead to a slow or complete failure of the learning process. Therefore, this study finalized the configuration of the reward values in Eq 3.1 through a series of systematic experiments and parameter adjustments. These adjustments were made to significantly improve the stability and generalization ability of the algorithm.

where ${x}_{a}, {y}_{a}, {z}_{a}$ represent the position coordinates of the end of the manipulator at the current time, and ${x}_{o}, {y}_{o}, {z}_{o}$ represent the position coordinates of the target object.

where ${x}_{b}, {y}_{b}, {z}_{b}$ represent the end position coordinates of the manipulator at the previous time, and ${x}_{o}, {y}_{o}, {z}_{o}$ represent the position coordinates of the target object.

Each time the manipulator interacts with the environment and moves one step, it will receive a reward or punishment according to the distance relationship between ${d}_{ao}$ and ${d}_{bo}$. When ${d}_{ao} < {d}_{bo}$, it means that the manipulator moves toward the target, and a reward is given to the manipulator; when ${d}_{ao} = {d}_{bo}$, it indicates that the manipulator is in a stagnant state, and a little punishment will be given; when ${d}_{ao} > {d}_{bo}$, it indicates that the manipulator moves in the wrong direction, and a big punishment will be given at this time.

(2) Core area acceleration guidance reward

The core area acceleration guidance reward is designed to distribute rewards according to the importance of the area where the manipulator is located.

where ${r}_{2}$ is the core area acceleration guidance reward, provided that the end-effector of the manipulator has not exceeded the motion execution space and the maximum number of steps is not exceeded. Based on the 7-degree-of-freedom KUKA manipulator with an arm length of 0.8 m, a workspace size of [0.5, 0.6, 0.55], and a target object size of [0.025, 0.1, 0.025], three different distance thresholds are designed in Eq. 3.4 to allow the control system enough time and space to make adjustments for finer control over the process of approaching the target object. The core area acceleration guidance reward is divided into three different regions, and different functional forms are used to incrementally increase or decrease the reward. The outermost region indicates that when the coordinate distance between the manipulator end-effector and the target object ranges from 0.04 to 0.1, the rewards are distributed through a linear function. The middle region indicates that when the coordinate distance between the manipulator end-effector and the target object ranges from 0.01 to 0.04, the rewards are distributed through the composite function composed of linear and exponential functions, which accelerates rewards. The core region indicates that when the coordinate distance range between the manipulator end-effector and the target object is less than or equal to 0.01 (belonging to the range of tracking success determination), a larger reward is added to the composite function to enhance guidance.

(3) Ladder adaptability reward

where ${r}_{3}$ denotes the ladder adaptability reward, which is designed to make the manipulator track the target object faster. When the manipulator enters into the reward range of the core region of Eq 3.4, the reward is given at this moment according to the cumulative length step from the initial to the tracking success. Equation 3.5 adds a variable $r\_step$ set to a fixed value of 20, mainly to introduce a baseline value so that the manipulator will be rewarded positively for completing the task within the first 20 steps and negatively otherwise. Therefore, the smaller the step, the larger the reward; the larger the step, the smaller the reward.

(4) Abnormal termination penalty

where ${r}_{4}$ is the abnormal termination penalty. When the manipulator exceeds the tracking range and the maximum tracking step, penalty reward $R\_p$ is given, and the round is terminated. In order to accelerate the convergence of the agent, when the manipulator exceeds the tracking range and the maximum tracking step, the parameter $R\_p$ in Eq 3.6 is set to -13. The value of this parameter has been experimentally derived to enable the manipulator to complete the task quickly, preventing wasted time. Then, the final fusion reward is calculated as follows:

where $r$ is the fusion reward. The pseudocode of the improved PPO algorithm is formulated in Algorithm 1.

4.   Experimental test and analysis

In order to evaluate the performance of the improved PPO algorithm, a 7-degree-of-freedom Kuka manipulator simulation experiment was set up on the PyBullet platform. The experimental environment was based on an Ubuntu operating system with an NVIDIA GeForce RTX 3090 and an Intel Gold 5218 processor, and the algorithm was written based on the PyTorch framework. In the whole simulation experiment setup, the size of the blue target block was [0.025, 0.1, 0.025], the field of view of the camera was 60° and the position was [0.59, 0, 0.8], and the white cube box on the desktop on the right side of the simulation environment (Figure 4) was designed by Gym to indicate the range of action of the manipulator, with a size of [0.5, 0.6, 0.55]. The target object randomly appears on different positions of the desktop within the action range of the manipulator. The manipulator uses the PPO algorithm to make a strategic decision based on the current environment state, selects the action with the highest expected reward value, and plans the motion trajectory accordingly. The main parameters of the PPO algorithm were set as in Table 1. A comparison of model training and testing under the two different reward guiding mechanisms (i.e., sparse and fusion reward) was performed. Experimental analysis was based on the tracking success rate and the average number of steps of the manipulator.

4.1.   Comparison of experimental results

Figures 5 and 6 show the iteration curves of the average episode reward of sparse reward and fusion reward in the training stage. The vertical axis (performance) refers to the average episode reward, and the horizontal axis refers to the complete number of interactions in the whole training stage. The sparse reward curve shows large fluctuations, especially in the late stage, which is due to the lack of effective guidance under the influence of the sparse reward; the manipulator can only search for the optimal strategy to track the target object through large-scale exploration, which is overall less stable. Compared with the sparse reward, the fluctuation amplitude of the fusion reward curve is significantly smaller than that of the sparse reward, and the overall convergence speed is relatively faster, indicating that the improved fusion reward can improve the convergence speed in the training phase.

Figure 7 shows the experimental comparison curve of the tracking success rate of sparse reward and fusion reward in the test phase, describing the relationship between the number of episodes and the success rate. Success rate refers to the proportion of success in 100 episodes, which is counted once every 100 episodes. The red dotted line is the test success rate of sparse rewards, and the blue one is the test success rate of fusion rewards. To satisfy the persuasiveness of the experimental analysis, the experimental data of 1000, 2000, 3000, and 4000 episodes were counted. Because the position of the tracked target object is random, both the sparse reward curve and the fusion reward curve have a certain volatility. A comparative analysis of the overall curve, as well as the specific values presented in Tables 2.1, 2.2, 2.3, and 2.4, reveals that, with the exception of a few discrete points, the success rate of fusion reward consistently surpasses that of sparse reward for the predominant part of the data. In addition, given that the manipulator needs to perform diverse trajectory tracking tasks based on the positions of different target objects, the fusion reward mechanism exhibits higher robustness and adaptability, which can guide the manipulator to complete the tracking task more accurately and thus significantly improve the tracking success rate.

Figure 8 shows the comparison curve of the average steps when the sparse reward and fusion reward track successfully in the test phase, describing the relationship between the average steps and episodes when the track succeeds. The average_step in the vertical axis refers to the cumulative steps for successful tracking in 100 episodes divided by 100. The average steps can effectively show the tracking efficiency of the manipulator. The smaller the average steps, the higher the tracking efficiency. The red dotted line describes the average steps of sparse reward, while the blue line describes the average steps of fusion reward. To satisfy the persuasiveness of the experimental analysis, the experimental data of 1000, 2000, 3000, and 4000 episodes were analyzed. It can be clearly observed through the dotted line that the tracking steps of the fusion reward are always located below the fusion reward, and the difference between the two is very large. The results show that the average number of steps of the sparse reward is above 22, while the average number of steps of the fusion reward is significantly lower, fluctuating between 8 and 9. A smaller average number of steps indicates that the shorter the time needed to track the target object, the more efficient the tracking is. Compared to the sparse reward, the tracking efficiency of the fusion reward is two times that of the sparse reward.

4.2.   Experimental data analysis

Table 3 shows data related to the tracking success rate of sparse rewards and fusion rewards in different ranges. The statistical indicators include mean, variance, maximum, minimum, and t-statistics analyzed by t-statistics and p-values. The statistical range covers 1000, 2000, 3000, and 4000 episodes. To facilitate the statistical analysis of data, mean data are kept at two decimal places (rounded), and variance data are kept with three significant figures (expressed in scientific notation). According to Table 3, compared with sparse rewards, the mean, maximum, and minimum values of tracking accuracy of fusion rewards in different ranges are increased, and the variance is greatly reduced. It can be concluded that fusion rewards not only have a higher tracking success rate than sparse rewards but also a higher tracking stability.

Table 4 shows the data related to the average number of steps for sparse and fusion rewards over different ranges. The statistical indicators include mean, variance, maximum, minimum, and t-statistics analyzed by t-statistics and p-values. The statistical ranges include 1000, 2000, 3000, and 4000 sets. According to the results in Table 4, the mean, maximum, and minimum values of the average set lengths of the fusion rewards in different ranges are significantly smaller, and the variance is greatly reduced, compared to the sparse rewards. Under the same conditions, the time complexity of trajectory planning with fused rewards over sparse rewards, in the worst case, is O($T$) and O(${T}^{2}$), respectively, where $T$ denotes the number of planning steps, indicating that fused rewards are faster and more stable than sparse rewards for tracking. Data analysis of variance, mean, maximum, and minimum of tracking success rate and average steps shows that compared with sparse rewards, fusion rewards have a higher and more stable tracking success rate, and when tracking is successful, fusion rewards have smaller and more stable tracking steps.

4.3.   Comparison of control performance

In Figure 9, the success rate curves of the various control algorithms are depicted for each 100 rounds of testing. The purple, red, blue, and green colors represent the improved PPO algorithm, the original PPO algorithm, the DDPG (deep deterministic policy gradient) algorithm, and the TD3 (trust region policy optimization with deep deterministic policy) algorithm, respectively. It is evident that the success rates of the original PPO algorithm and TD3 algorithm exhibit significant fluctuations, while the DDPG algorithm demonstrates a higher success rate and reduced variability compared to the previous two algorithms. In contrast, the improved PPO algorithm demonstrates a 94.88% success rate, which significantly surpasses the performance of the original PPO algorithm, the DDPG algorithm, and the TD3 algorithm. This enhancement signifies the enhanced stability of the manipulator trajectory planning control. Figure 10 illustrates the time required for successful planning for each algorithm in the test round. As shown, the improved PPO algorithm takes only 1.71 s to complete successful planning, a performance that outperforms the other three compared algorithms, reflecting a shorter planning time. Therefore, the improved PPO algorithm not only accelerates the planning process but also significantly improves the control performance of the robot arm through higher accuracy.

5.   Conclusions

The sparse reward mechanism in the PPO algorithm often leads the agent to collect many ineffective and expected samples during the sampling stage, which causes its policy optimization to update and converge slowly or even not converge. To overcome the influence of sparse rewards, this paper designs a fusion reward mechanism PPO algorithm and applies it to the manipulator target tracking. The experimental results show that the average tracking success rate of the PPO algorithm with fused reward mechanism is as high as 94.88%, and the average number of steps is about 9.7. The algorithm significantly improves the tracking accuracy and efficiency of the spatial manipulator and further enhances the stability of the manipulator in target-tracking tasks. However, the applicability of the algorithm on other types of manipulators remains to be verified, especially in terms of considering the effects of hardware differences, environmental adaptability, data requirements, and real application environments. The improved method proposed in this paper not only enables the manipulator to be better adapted to the traditional industrial automation field but also has the potential to be widely applied to the fields of healthcare, service robotics, and agriculture, thus enhancing the automation level and work efficiency in various industries.

Author contributions

Ruyi Dong: Conceptualization, Software, Investigation, Methodology, Validation, Writing ‒ original draft, Writing ‒ review & editing; Kai Yang: Conceptualization, Writing ‒ original draft, Writing ‒ review & editing; Tong Wang: Conceptualization, Writing ‒ original draft, Writing ‒ review & editing. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

This research was supported by the Science & Technology Development Project of Jilin Province, China (YDZJ202201ZYTS555).

Use of AI tools declaration

The authors declared they have not used Artificial Intelligence (AI) tools in the creation of this article.

Conflict of interest

The authors declare that there is no conflict of interest in this paper.