Figure 1.
The digitization process for fitness equipment.
Research on the relationship between drugs and targets is the key to precision medicine. Ion channel is a kind of important drug targets. Aiming at the urgent needs of corona virus disease 2019 (COVID-19) treatment and drug development, this paper designed a mixed graph network model to predict the affinity between ion channel targets of COVID-19 and drugs. According to the simplified molecular input line entry specification (SMILES) code of drugs, firstly, the atomic features were extracted to construct the point sets, and edge sets were constructed according to atomic bonds. Then the undirected graph with atomic features was generated by RDKit tool and the graph attention layer was used to extract the drug feature information. Five ion channel target proteins were screened from the whole SARS-CoV-2 genome sequences of NCBI database, and the protein features were extracted by convolution neural network (CNN). Using attention mechanism and graph convolutional network (GCN), the extracted drug features and target features information were connected. After two full connection layers operation, the drug-target affinity was output, and model was obtained. Kiba dataset was used to train the model and determine the model parameters. Compared with DeepDTA, WideDTA, graph attention network (GAT), GCN and graph isomorphism network (GIN) models, it was proved that the mean square error (MSE) of the proposed model was decreased by 0.055, 0.04, 0.001, 0.046, 0.013 and the consistency index (CI) was increased by 0.028, 0.016, 0.003, 0.03 and 0.01, respectively. It can predict the drug-target affinity more accurately. According to the prediction results of drug-target affinity of SARS-CoV-2 ion channel targets, seven kinds of small molecule drugs acting on five ion channel targets were obtained, namely SCH-47112, Dehydroaltenusin, alternariol 5-o-sulfate, LPA1 antagonist 1, alternariol, butin, and AT-9283.These drugs provide a reference for drug repositioning and precise treatment of COVID-19.
Citation: Xianfang Wang, Qimeng Li, Yifeng Liu, Zhiyong Du, Ruixia Jin. Drug repositioning of COVID-19 based on mixed graph network and ion channel[J]. Mathematical Biosciences and Engineering, 2022, 19(4): 3269-3284. doi: 10.3934/mbe.2022151
| [1] | Hong Yuan, Jing Huang, Jin Li . Protein-ligand binding affinity prediction model based on graph attention network. Mathematical Biosciences and Engineering, 2021, 18(6): 9148-9162. doi: 10.3934/mbe.2021451 |
| [2] | Saranya Muniyappan, Arockia Xavier Annie Rayan, Geetha Thekkumpurath Varrieth . DTiGNN: Learning drug-target embedding from a heterogeneous biological network based on a two-level attention-based graph neural network. Mathematical Biosciences and Engineering, 2023, 20(5): 9530-9571. doi: 10.3934/mbe.2023419 |
| [3] | Dong Ma, Shuang Li, Zhihua Chen . Drug-target binding affinity prediction method based on a deep graph neural network. Mathematical Biosciences and Engineering, 2023, 20(1): 269-282. doi: 10.3934/mbe.2023012 |
| [4] | Lei Chen, Kaiyu Chen, Bo Zhou . Inferring drug-disease associations by a deep analysis on drug and disease networks. Mathematical Biosciences and Engineering, 2023, 20(8): 14136-14157. doi: 10.3934/mbe.2023632 |
| [5] | Ze-Yu Zhang, Zhu-Jun Mao, Ye-ping Ruan, Xin Zhang . Computational identification of Shenshao Ningxin Yin as an effective treatment for novel coronavirus infection (COVID-19) with myocarditis. Mathematical Biosciences and Engineering, 2022, 19(6): 5772-5792. doi: 10.3934/mbe.2022270 |
| [6] | Deepalakshmi Sarkarai, Kalyani Desikan . QSPR/QSAR analysis of some eccentricity based topological descriptors of antiviral drugs used in COVID-19 treatment via Dε- polynomials. Mathematical Biosciences and Engineering, 2023, 20(9): 17272-17295. doi: 10.3934/mbe.2023769 |
| [7] | Mingju Chen, Sihang Yi, Mei Yang, Zhiwen Yang, Xingyue Zhang . UNet segmentation network of COVID-19 CT images with multi-scale attention. Mathematical Biosciences and Engineering, 2023, 20(9): 16762-16785. doi: 10.3934/mbe.2023747 |
| [8] | Huiqing Wang, Sen Zhao, Jing Zhao, Zhipeng Feng . A model for predicting drug-disease associations based on dense convolutional attention network. Mathematical Biosciences and Engineering, 2021, 18(6): 7419-7439. doi: 10.3934/mbe.2021367 |
| [9] | Ali Al Khabyah, Muhammad Kamran Jamil, Ali N. A. Koam, Aisha Javed, Muhammad Azeem . Partition dimension of COVID antiviral drug structures. Mathematical Biosciences and Engineering, 2022, 19(10): 10078-10095. doi: 10.3934/mbe.2022471 |
| [10] | Shizhen Huang, ShaoDong Zheng, Ruiqi Chen . Multi-source transfer learning with Graph Neural Network for excellent modelling the bioactivities of ligands targeting orphan G protein-coupled receptors. Mathematical Biosciences and Engineering, 2023, 20(2): 2588-2608. doi: 10.3934/mbe.2023121 |
Research on the relationship between drugs and targets is the key to precision medicine. Ion channel is a kind of important drug targets. Aiming at the urgent needs of corona virus disease 2019 (COVID-19) treatment and drug development, this paper designed a mixed graph network model to predict the affinity between ion channel targets of COVID-19 and drugs. According to the simplified molecular input line entry specification (SMILES) code of drugs, firstly, the atomic features were extracted to construct the point sets, and edge sets were constructed according to atomic bonds. Then the undirected graph with atomic features was generated by RDKit tool and the graph attention layer was used to extract the drug feature information. Five ion channel target proteins were screened from the whole SARS-CoV-2 genome sequences of NCBI database, and the protein features were extracted by convolution neural network (CNN). Using attention mechanism and graph convolutional network (GCN), the extracted drug features and target features information were connected. After two full connection layers operation, the drug-target affinity was output, and model was obtained. Kiba dataset was used to train the model and determine the model parameters. Compared with DeepDTA, WideDTA, graph attention network (GAT), GCN and graph isomorphism network (GIN) models, it was proved that the mean square error (MSE) of the proposed model was decreased by 0.055, 0.04, 0.001, 0.046, 0.013 and the consistency index (CI) was increased by 0.028, 0.016, 0.003, 0.03 and 0.01, respectively. It can predict the drug-target affinity more accurately. According to the prediction results of drug-target affinity of SARS-CoV-2 ion channel targets, seven kinds of small molecule drugs acting on five ion channel targets were obtained, namely SCH-47112, Dehydroaltenusin, alternariol 5-o-sulfate, LPA1 antagonist 1, alternariol, butin, and AT-9283.These drugs provide a reference for drug repositioning and precise treatment of COVID-19.
Physical training, an integral aspect of modern life, encompasses various systematic and targeted exercise activities aimed at improving one's physical fitness, overall health and well-being [1]. The significance of physical training extends beyond individual benefits, as it not only helps to prevent various health issues such as obesity, cardiovascular diseases, and diabetes, but it also enhances mental health, boosts self-esteem and fosters social connections [2]–[5]. From a societal perspective, promoting physical training contributes to reduced healthcare costs, increased productivity and improved quality of life, making it a vital factor in the well-being of communities and nations alike [6]. As the demand for efficient and personalized physical training methods grows, there is an emerging need to leverage advanced technologies to optimize training outcomes and maximize the benefits derived from exercise.
Physical training has evolved significantly. Figure 1 illustrates the technological advancements in fitness equipment over time. In the 18th and early 19th centuries, physical training primarily served military purposes, emphasizing the development of physical prowess for combat readiness [7]. The inception of the modern Olympic Games in the early 20th century marked a turning point, as various forms of exercise and fitness training emerged, including aerobic exercise, strength training, yoga and Pilates [8]. The advent of modern technology has further revolutionized physical training, employing tools such as a smart bracelet and intelligent body fat scales [9]. However, current physical training methodologies face notable challenges. Most devices on the market are limited to single-modality monitoring, providing an incomplete understanding of an individual's overall physical condition [10]–[12]. Addressing this gap, multi-modal sensor fusion systems that combine data from various sensors may serve as solutions to provide a comprehensive picture of an individual's fitness [13]. These systems have the potential to significantly enhance the effectiveness of physical training by offering more accurate, personalized feedback and recommendations based on a holistic understanding of the trainee's physical state.
In light of these challenges, the primary objective of this research was to validate the effectiveness of the proposed multi-modal sensor fusion system in enhancing physical training performance. The developed system integrates a diverse range of sensors, including inertial measurement units (IMUs), heart rate sensors, electromyography (EMG) sensors, pressure sensors, GPS sensors, cameras, computer vision systems and environmental sensors. The research methodology here combines principles from geometry, biology and kinematics to develop a comprehensive understanding of an individual's physical state and performance during training. Figure 2 presents the design idea of the system and the core technologies involved.
This study had the following goals. First, we introduce a multi-modal sensor fusion system that revolutionizes the approach to physical training. Our system has been designed, implemented and thoroughly evaluated to provide a distinctive and innovative approach, surpassing the capabilities of conventional single-modality systems. Integrating multiple sensors enables the acquisition of richer and more comprehensive data and insights, facilitating a deeper understanding of an individual's physical condition during training.
Second, the system incorporates advanced sensor technology and employs sophisticated data processing algorithms. This integration ensures the system's ability to monitor various aspects of an individual's physical condition with high accuracy and reliability. Through the utilization of cutting-edge sensors and advanced algorithms, we enhance the precision of data collection and analysis, enabling more effective monitoring of the trainee's physical state.
Last, the proposed multi-modal sensor fusion system showcases its potential to significantly enhance physical training outcomes. By leveraging the comprehensive data and insights obtained from multiple sensors, the system provides tailored feedback and recommendations. This personalized approach, rooted in a comprehensive understanding of the trainee's fitness level and performance, has the potential to optimize training routines and unlock improved outcomes in physical training.
In essence, this paper delves into the utilization of multi-modal sensor fusion systems for physical training. The subsequent sections are organized as follows. Section 2 provides an extensive literature review, examining prior work in the field. Building upon this foundation, section 3 outlines the proposed methodology, elucidating the system design. In Section 4, the implementation of our multi-modal sensor fusion system is presented. Section 5 verifies the effectiveness of our approach through experimental demonstrations. Finally, Section 6 concludes the paper by summarizing the principal contributions and suggesting future research directions.
Within the realm of physical training, various single-modality monitoring approaches have been utilized, such as force sensors, depth cameras, IMUs and heart rate monitors, to track different facets of human performance. However, these individual methods are not without their limitations, including decreased precision and unreliable data. To address these shortcomings, a wearable sensing system was developed to enhance rehabilitation for neurological patients by employing an electronic glove equipped with compact force sensors and a modified interface to measure fingertip force [14]. Additionally, a novel teaching methodology was introduced, employing a gamified learning process that promotes physical activity through the use of interactive balls in educational activities [15]. Notably, this system utilized a Kinect depth camera for image recognition, enabling the detection of ball hits on activity projections. In [16], to prevent cognitive and physical decline in older individuals with mild cognitive impairment, an exergame was designed. This exergame employed IMUs worn on the wrists and feet of the participants to track their movements as they navigated through the game environment. Within the field of rehabilitation training, Batalik et al. [17] explored the potential of telerehabilitation with wrist heart rate monitors as an alternative to traditional outpatient cardiac rehabilitation. The study demonstrated significant improvements in physical fitness for both regular outpatient training and interventional home-based telerehabilitation groups. Most recently, Bo and Sun [18] developed a heart-rate monitoring platform for college sports training, utilizing wireless networks. This platform featured multi-level link equalization, a transmission numerical model for heart-rate monitoring channels, data redundancy verification and a topology stage structure.
The utilization of multi-modal sensor fusion systems has garnered considerable attention in diverse domains, including affective computing, robotics and automated driving [19], [20]. These systems combine data from multiple sensors to provide a more comprehensive understanding of various phenomena. A crucial aspect of our research is the application of such systems in biomedical engineering, especially in enhancing physical training. In the domain of wearables, a wearable multi-modal biosensing system has been developed to collect, synchronize, record and transmit data from diverse biosensors, such as systems of PPG, EEG, eye-gaze headset, body motion capture and GSR [21]. The performance of these sensors was evaluated by comparing them to standard research-grade biosensors. Comparatively, a study [22] yielded a scalable design model for artificial fingers in anthropomorphic robotic and prosthetic hands. This design integrated mechanical components, embedded electronics and a multi-modal sensor system. The fully parametric design enabled automated scaling, and interchangeable electronic modules facilitated mechanical adjustments. In the context of urban autonomous driving, Marco et al. [23] proposed a multi-modal sensor fusion scheme for accurately estimating 3D vehicle velocity, pitch and roll angles. The method simultaneously estimated gyroscope and accelerometer biases to enhance accuracy, proving itself to be effective during regular urban drives and collision avoidance maneuvers.
A multitude of multi-modal sensor fusion systems have emerged, aiming to augment physical training outcomes and yield an extensive comprehension of human performance. These systems amalgamate diverse sensor types, such as GPS, heart-rate monitor, IMUs and motion-capture devices, to capture a broad range of physical activity aspects. Noteworthy contributions include Ma and Hu's development [24] of a home-oriented cyber-physical system that enhances motion coordination capabilities via physical training. Their system employs a thermal camera to record leg and foot motions, as well as insole pressure sensors for plantar pressure measurement. Innovative algorithms for leg skeleton extraction and motion signal auto-segmentation, recognition and analysis were devised.
As early as 2016, Torreño et al. [25] conducted an investigation by utilizing GPS and heart-rate technology to explore the match running profiles, heart rates and performance efficiency indices of professional soccer players during official club-level games. Similarly, the utilization of a Polar GPS device alongside heart-rate sensors allowed for the determination of motion intensity in physical education settings [26]. To address the rehabilitation requirements, a novel virtual personal trainer system, tailored for rehabilitation purposes, was proposed. This system employed nine IMU sensors and a 3D camera for comprehensive full-body motion tracking [27]. The incorporation of a 3D camera mitigated IMU sensor output instability, counteracted gyroscopic drift and reduced electromagnetic interference. The resulting wireless full-body sensor array system provided a cost-effective alternative to on-site trainers. Stroke rehabilitation-focused physical training involved the development of a portable rehabilitation platform, encompassing both mental and physical training modalities [28]. This platform encompassed an EEG-based BCI system for mental training, a force sensor-embedded orthosis for elbow extension/flexion and an FES unit for hand extension. Later, a group [29] proposed a data fusion algorithm that combined skeletal data from three Kinect devices to overcome the limitations of single-device usage for human skeletons and motion tracking within sports medicine and rehabilitation. This approach offered comprehensive 3D spatial coverage of subjects and demonstrated an enhanced tracking accuracy improvement of 15.7%. As multi-modal sensor fusion technology advances, substantial strides are anticipated in the areas of data fusion algorithms and machine learning techniques, elevating the capabilities of these systems even further [30].
Multi-modal sensor fusion systems play a pivotal role in advancing research and technology [31]. Researchers have focused on various aspects of these systems, such as PRF-PIR, unified camera tracking approaches and novel methods for human activity recognition. In the realm of PRF-PIR, Yuan et al. [32] proposed a passive, multi-modal sensor fusion system named PRF-PIR. This system consists of a software-defined radio device and a novel passive infrared (PIR) sensor system. Using a recurrent neural network (RNN) as the human identification and activity recognition model, the PRF-PIR system accurately and non-intrusively monitors human activity in indoor environments. The system's effectiveness was validated through data collection from 11 activities performed by 12 human subjects, employing explainable artificial intelligence methodologies. In [33], Oskiper et al. came up with a unified approach for a camera tracking system, employing an error-state Kalman filter algorithm. This approach utilizes relative and global measurements obtained from image-based motion estimation, landmark matching and radio frequency-ranging radios. The proposed approach demonstrated long-term stability and overall accuracy under vision-aided and vision-impaired conditions, as evidenced by the rendering of views from a 3D graphical model and actual video images. Regarding human activity recognition, an efficient method that fuses data from inertial sensors (IMUs), surface EMG, and visual depth sensors have been developed. This approach exhibits superior performance compared to single- or dual-sensor approaches, with high robustness against occlusions, data losses and other non-structured environmental events [34].
The integration of artificial intelligence into sensor fusion technology, encompassing data fusion algorithms and machine learning methods, holds immense potential for advancements [35]–[37]. Xu and his colleagues [38] posited that a self-adaptive wavelet transform-based data fusion algorithm, suitable for both static and dynamic systems, can achieve an optimal estimation of a measurand with minimum mean square error in multisensor systems. Similarly, the GAPSOBP algorithm, which intelligently combines a BP neural network, genetic algorithm and particle swarm optimization algorithm, exhibits efficiency in reducing the volume of data transmitted to a base station or sink node within wireless sensor networks, thus conserving network energy [39]. In the field of physical training, the utilization of data fusion algorithms and machine learning methods has substantial value. They enable the development of multi-modal sensor fusion systems that yield richer data and insights than those of conventional single-modality systems [40]. Furthermore, the synergy between multi-modal sensor fusion systems and virtual reality (VR) presents an opportunity to enhance various applications. Through the integration of data from diverse sensors, multi-modal sensor fusion systems offer a comprehensive and precise understanding of the physical environment and human behavior. In contrast, VR creates immersive and interactive virtual environments that simulate real-world scenarios, allowing users to engage with virtual objects and surroundings. The combination of these technologies results in a robust and authentic training or simulation platform, providing users with an immersive, interactive experience that delivers accurate and comprehensive feedback [41]–[43].
Table 1 provides a comprehensive overview of the diverse sensing modalities employed in mobile and wearable sensor-based multi-modal applications.
| Applications | Refs. | Year | Multi-sensor | Methods |
| Neurologic Rehabilitation Trainings | [14] | 2018 | FSR, Wearable+force+embedded sensors | Rigid discs Electronic circuit |
| Physical Activity | [15] | 2021 | Kinect sensor ECG |
MOCAP TEL |
| Heart-Rate Monitor | [17] | 2020 | Wearable sensors | Polar Flow web |
| [18] | 2023 | Temperature+heart rate+acceleration+smoke sensors | Wireless Network | |
| Bio signals | [21] | 2018 | PPG EEG Body motion capture GSR Inertial sensors |
SSVEP |
| Vehicle Motion State Estimation | [23] | 2020 | ESC GNSS IMU+Series-grade chassis sensors |
Intrinsic Euler angles Immersion transformation |
| Human Identification and Activity | [32] | 2022 | PIR PRF-PIR IMU MI-PIR |
RGBD CNN NBC FMCW RNN |
| Infrastructure-free Localization in Vision-impaired Environments | [33] | 2010 | EO IMU RF |
HMD Kalman filter |
DownLoad:
CSV
This section presents the proposed methodology, expounding on the intricate configuration and execution of our multi-modal sensor fusion system. In Figure 3, an overview of the system's use in physical training is depicted. In the following subsections, we will describe each module in detail.
1. Skeleton tracker (Section 3.1), which integrates depth detection, bone tracking, face recognition and voice recognition using diverse cameras.
2. Body angle measurement (Section 3.2), which utilizes specialized equipment to collect EMG signals from various arm segments, providing real-time display and raw data export.
3. Finger motion monitor (Section 3.3), which captures real-time movement data from individual hand joints, utilizing non-sensing wear technology and facilitating data transfer.
4. VR headset (Section 3.4), allowing users to immerse themselves in a VR environment with the freedom to navigate and observe in any direction.
The skeleton tracker's primary function in this research is to construct an accurate and dynamic model of the user's body motion during physical training. It uses multiple cameras and chips to detect depth, track bones and recognize faces and voices. Meanwhile, it allows for high-resolution output (1920 × 1080, 30 fps in color; 512 × 424, 30 fps in depth) with remarkable accuracy. Depth detection relies on the projection of infrared rays through the infrared camera to generate reflected light, which facilitates the determination of an object's position and the creation of a depth image based on spare flight time. Skeleton tracking was primarily designed to construct a bone map for up to six players and accurately track the corresponding bone nodes throughout the physical training session.
The body angle measurement apparatus was designed to capture the subtleties of body movement, focusing primarily on the arm segment, by using EMG signals. It gives real-time data on muscle activity during training. The EMG signal collector primarily comprises a wireless collector, a wireless EMG sensor (one channel) and a wireless biaxial joint goniometer (two tracks). In our system, the Biometrics GZ12 EMG signal collector records electrical muscle activity in response to different physical actions. This information is then exported for in-depth analysis, allowing us to examine the efficiency of the training, identify potential strain or injury risks and suggest improvements in a training method.
Accurate measurement and analysis of hand movements necessitates sophisticated algorithms and advanced sensors. The finger motion monitor, represented here by our data glove, is a critical component that captures nuanced data about hand joint movements during physical training. Equipped with 9-axis MEMS inertial sensors and a vibration feedback module, the glove captures and communicates detailed hand motion data. By employing inverse dynamics, the glove accurately reconstructs bone motion, enabling the faithful reproduction of natural movement in virtual environments. Furthermore, the palm of the glove incorporates a built-in vibration feedback module, which triggers vibration effects corresponding to different scenarios, thereby enhancing the immersive experience in computer designs. To ensure seamless communication, the gloves utilize 24-GHz wireless transmission, achieving a high frame rate of 120 Hz for one hand and 240 Hz for both hands. This high frame rate, coupled with a low delay transmission effect within 10 ms, ensures smooth and responsive interactions. In this research context, we used it to understand the involvement of hand and finger movements during the exercise process. These data are particularly valuable for the evaluation of the precision and accuracy of hand movements during training, allowing us to devise more effective training regimens or corrective measures as required.
The Pico 4 VR headset is our system's main user interface. The headset provides a fully immersive VR environment, allowing the user to perform physical training actions in a controlled and responsive setting. The Pico 4 is a standalone headset with six-degree-of-freedom inside-out tracking, the weighing 450 g. It prioritizes comfort with an adjustable head strap and face cushion, offering a smooth and immersive experience with a 101-degree FOV and 75 Hz refresh rate. Powered by a Qualcomm Snapdragon 845 processor, it has 4 GB of RAM and 128 GB of storage. The Pico 4 includes built-in stereo speakers, a 3.5 mm audio jack, two six-degree-of-freedom controllers for intuitive interaction, WiFi 6, Bluetooth 5.0 and hand-tracking capabilities, and it runs on the proprietary PICOS. In summary, the Pico 4 VR headset offers immersive experiences without external sensors or cameras. It is well suited for enhancing physical training and facilitating engaging virtual environments.
Our hand-tracking system represents a significant advancement in the field of the precise tracking of complex hand movements (Figure 4). Unlike traditional systems that are confined to a fixed workspace, ours allows users to move freely while wearing the equipment, thus introducing a more natural interactive user experience.
The system comprises two individual subsystems for each hand, each utilizing a glove equipped with a constellation of sensors. Each glove houses six nine-axis IMUs and five flex sensors, providing a total of 12 IMUs and 10 flex sensors for both hands. The IMUs incorporate a gyroscope, accelerometer and magnetometer with measurement ranges of ±2000 dps, ±16 g and ±8.1 Ga, respectively. These sensors work in unison to provide accurate tracking of the hand's spatial orientation and acceleration, as well as the earth's magnetic field, to determine directional heading.
The flex sensors, strategically positioned, monitor the bending of the fingers, thereby capturing even the subtlest of hand movements. These sensors are calibrated to deliver a static accuracy of 0.2° RMS for roll/pitch and a dynamic accuracy of 1.0° RMS. The angular measurement resolution stands at an impressive 0.02°.
The gloves employ dual-band (2.4 GHz/5.8 GHz) wireless communication technology, achieving a synchronization accuracy of 10 µs, an industry-leading figure. Furthermore, the system's data communication supports manual and automatic channel switching, thereby mitigating the influence of potential wireless interference in the surroundings.
The system's hardware components are engineered for compatibility with 100BASE-T interfaces and the IEEE 802.3 af/at standards for Power over Ethernet, making it a versatile solution for various user requirements.
Further enhancing the tracking capabilities of our system is a Microsoft Kinect V2, incorporated for the capture of body skeleton data and recognition of overall body movements. The amalgamation of hand movement data from our custom gloves and body movement data from Kinect V2 results in a comprehensive and seamless body tracking experience for the user.
Augmenting the sensor suite of each glove is a wireless EMG data collector from Biometrics. This collector gathers critical data pertaining to forearm muscle activity, providing another data source for enhanced movement recognition. Our approach of extensive sensor fusion, utilizing data from multiple sensor types, allows the system to achieve an unprecedented level of precision in hand movement recognition.
Powering our tracking system is the Robot Operating System (ROS), which was run on an Ubuntu 20 machine. This computer communicates with the gloves via a high-speed wireless router, ensuring minimal latency and maximum throughput in data transmission. Furthermore, MATLAB was employed to process the collected data, which were subject to interaction in real-time within a VR scene in the PICO 4 system. This VR scene, developed in Unity, maintains a real-time communication link with MATLAB, thus enabling fluid interaction between the user and the virtual environment.
Given its precision, our system is optimal for intricate physical training regimens, including those that emphasize precise hand movements such as rehabilitation exercises. It is invaluable for trainers seeking detailed feedback on a trainee's form, grip and finger positioning.
| Device/Software | Version/Parameters |
| Inertial Sensor | Gyroscope: ±2000 dps, Accelerometer: ±16 g, Magnetometer: ±8.1 Ga |
| Flex Sensor | 10 (5 per hand) |
| Kinect V2 | Body skeleton tracking |
| EMG Data Collector | Biometrics |
| Communication Technology | Dual-band (2.4 GHz/5.8 GHz), Sync accuracy: 10 µs |
| ROS | Melodic Morenia (1.14.11) |
| Ubuntu | 20.04 LTS |
| MATLAB | R2023a |
| PICO 4 VR System | Latest firmware as of June 2023 |
| Unity | 2023.1.2 |
DownLoad:
CSV
In conclusion, our system stands out due to its detailed hand tracking, full-body motion capture and virtual environment integration. It provides a flexible solution that fits various use cases.
First, we capture the basic posture of the human body through the skeleton tracking feature of the Kinect V2. For each joint, Kinect V2 provides its position in 3D space. To compute the angle between two joints, we can use the dot product of vectors formed by the joints, which is calculated as follows:
where
Kinect provides the 3D positions of various joints in the human body. Each joint Ji is defined in a 3D Cartesian space, and thus has a position vector in this space denoted by
Now, consider three joints J1, J2 and J3 in sequence, where J2 is the joint of interest and J1 and J3 are adjacent joints. The vectors formed from J2 to J1 and J2 to J3 are respectively denoted by A and B, which can be calculated as follows:
The angle θ between these two vectors, which corresponds to the angle of the joint J2, can be calculated by using the dot product of A and B:
where ||A|| and ||B|| are the magnitudes of vectors A and B respectively.
The glove is furnished with inertial sensors and the flex sensors, capturing the movements of fingers. The fusion of data from these sensors allows us to achieve accurate finger movement estimation. The IMU sensors operate at a sampling rate of 200 Hz, and flex sensors operate at a rate of 100 Hz. All sensors are synchronized via a central microcontroller that timestamps the data before passing it on for processing.
We leverage an adaptive neural network algorithm to process the sensor data and predict hand movements. It helps to overcome the limitations of traditional methods, like sensitivity to the initial conditions and the constraints of linear systems, thus providing robust and precise hand tracking.
The flex sensors in the glove measure the amount of bend in each finger joint. For each joint Fi we have a corresponding bending measurement
The IMU provides data for the orientation of the glove (and, consequently, the hand and the fingers) in the form of quaternion values
For precise measurement of the finger joint angle
Where Kk is the Kalman gain, zk is the actual sensor reading, H is the transformation matrix, I is the identity matrix,
The bending measurement
In the pursuit of optimal sensor fusion, algorithm selection is crucial. For this research, the gated recurrent unit-RNN has been employed to predict the finger joint angles based on the sequential sensor readings. The primary reason for adopting the GRU-RNN model over other deep learning or traditional algorithms lies in its inherent capabilities that are tailored for sequential data. The GRU makes use of two gating mechanisms, namely the reset gate and the update gate, which control the flow of information inside of the unit. The sensors' raw data feed into the GRU model at their respective sampling rates, with missing data points interpolated to ensure continuous input.
For a given sequence of sensor readings
where Wz, Wr are weight matrices, bz, br are bias terms and σ is the sigmoid function.
The candidate activation
Finally, the hidden state ht is updated as follows:
This hidden state ht represents the predicted joint angle at time step t.
A key property of GRUs (and RNNs in general) is that they have the ability to handle sequences of data and model temporal dependencies, which make them particularly suited for our application. However, the learning of GRUs involves the challenging task of learning long-term dependencies due to the vanishing gradients problem and requires careful initialization and potentially regularization methods to ensure stable convergence.
To interact with the PICO 4 VR system, we used Unity3D to develop a VR scene and establish communication with MATLAB. The human movement data processed by MATLAB are sent to Unity in real-time, which then updates the VR scene accordingly. This provides the user with an immersive experience as their physical movements are accurately reflected in the virtual environment, which is especially beneficial for physical exercise applications.
For future work, we intend to enhance the system by incorporating more advanced sensor fusion algorithms and neural networks, further improving the robustness and accuracy of human movement tracking in VR-based human-computer interaction. We also plan to expand its application areas to other domains, such as medical rehabilitation and professional sports training.
In order to evaluate the performance of the proposed GRU method and compare it with other algorithms, a comprehensive set of experiments were conducted. A dataset comprising time-series data from the motion-tracking system was prepared. This dataset consisted of sensor readings from the hand-tracking gloves, Kinect V2 and EMG data collectors.
Each time step in the data corresponds to a specific joint angle configuration for the human subject. The goal of the algorithms is to predict the joint angles at the next time step given the current and previous sensor readings. The ground truth for these predictions comes from the actual joint angles measured by the motion-tracking system.
The experiment involved two main steps: feature extraction and model training. The data from the Kinect and glove sensors were pre-processed and segmented into windows of 2 second with 50% overlap. For each window, a set of features were extracted, which included statistical features (mean, variance, skewness and kurtosis), frequency-domain features (FFT coefficients) and time-domain features (zero-crossings, peak values). These features were then used as input to the machine learning models.
The models were trained on 80% of the data, with the remaining 20% used for testing. The performance of each model was evaluated by using five metrics: accuracy, precision, recall, F1 score and average prediction time. The accuracy was calculated as the proportion of correctly predicted samples. The precision was calculated as the ratio of correctly predicted positive observations to the total predicted positive observations. The recall was calculated as the ratio of correctly predicted positive observations to all observations in the actual class. The F1 score was calculated as 2(Recall * Precision) / (Recall + Precision). The average prediction time was measured by averaging the time taken by the model to predict the joint angles for all samples in the test set.
The accuracy, precision, recall, and F1 score are popular metrics used in machine learning, and they were computed as follows:
The comparison of the GRU-RNN algorithm with other machine learning methods is summed up in Table 3. It is clear from the results that the GRU-RNN algorithm outperforms the other algorithms in terms of accuracy, precision, recall and F1 score. This implies that the GRU-RNN algorithm is more successful in terms of correctly predicting the joint angles from the sensor data. Furthermore, the GRU-RNN algorithm also demonstrates superiority in terms of the average prediction time, further enhancing its suitability for real-time applications. The superior performance of the GRU-RNN algorithm can be attributed to its ability to effectively capture the temporal dependencies in the sensor data, which is crucial for accurate joint angle prediction. This makes the GRU-RNN algorithm a robust and efficient choice for the motion-tracking system.
| Algorithm | Accuracy | Precision | Recall | F1 Score | Average Prediction Time (ms) |
| GRU-RNN | 0.97 | 0.98 | 0.96 | 0.97 | 4.5 |
| Feedforward Neural Network | 0.85 | 0.87 | 0.83 | 0.85 | 5.2 |
| Support Vector Machine | 0.81 | 0.83 | 0.79 | 0.81 | 6.3 |
| Random Forest | 0.76 | 0.77 | 0.75 | 0.76 | 7.8 |
DownLoad:
CSV
As observed in Table 3, our proposed method outperforms the other techniques in terms of accuracy, precision, recall and F1 score. Furthermore, it provides the best real-time processing capabilities, with a shorter computational time than other methods. Thus, our system's application of networks demonstrates superior performance, further bolstering its potential for precise, robust and real-time hand motion tracking in dynamic environments.
The bar chart provided in Figure 5 presents a comparison of the GRU-RNN algorithm with other common machine learning algorithms in terms of accuracy.
Evidently, the GRU-RNN outperforms the other algorithms in terms of accuracy. It is important to note that the superiority of the GRU-RNN is attributable to its inherent ability to capture temporal dependencies in sequential data. In the case of joint angle prediction, the current joint angles are closely related to the previous states due to the physical constraints of human body motion. This context is something that the GRU-RNN captures well.
The feedforward neural network, support vector machine (SVM) and random forest methods, though being competent machine learning models, are unable to handle sequential data as effectively. This is due to their inability to maintain a “memory” of previous inputs in the sequence.
The feedforward neural network, in this case, fails to consider the temporal sequence of data. On the other hand,the SVM and random forest work on the assumption that the data samples are independent, which is not the case with sequential sensor data. Hence, their performance falls short as compared to the GRU-RNN.
In conclusion, for tasks that involve sequential data, such as the joint angle prediction in our motion-capture system, the use of the GRU-RNN provides significant advantages over other methods in terms of accuracy. This finding is vital for the enhancement of the effectiveness and reliability of our system in order to deliver a more immersive VR experience for users.
We have successfully developed a multi-modal sensor fusion system that can enhance physical training by delivering detailed, real-time feedback on the user's performance. Through the use of diverse sensor sources including IMUs, flex sensors, EMG sensors and Microsoft's Kinect V2, our system can capture comprehensive body and hand motion data. Our results demonstrate that the GRU-RNN algorithm, as employed in our system, outperforms traditional machine learning algorithms in terms of accuracy, precision, recall and F1 score. Furthermore, the integration of our system with the PICO 4 VR environment offers an immersive and interactive experience, fostering user engagement in physical training. Unlike many traditional systems, ours overcomes the constraint of a fixed workspace, allowing users the flexibility to train in a free-form environment. Future work will focus on further improving the accuracy of motion estimation and enhancing the immersive experience of the VR environment.
The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.
| [1] |
K. Huang, T. Fu, L. M. Glass, M. Zitnik, C. Xiao, J. Sun, DeepPurpose: a deep learning library for drug–target interaction prediction, Bioinformatics, 36 (2020), 5545–5547. https://doi.org/10.1093/bioinformatics/btaa1005 doi: 10.1093/bioinformatics/btaa1005
|
| [2] |
T. U. Singh, S. Parida, M. C. Lingaraju, M. Kesavan, D. Kumar, R. K. Singh, Drug repurposing approach to fight COVID-19, Pharmacol. Rep., 72 (2020), 1479–1508. https://doi.org/10.1007/s43440-020-00155-6 doi: 10.1007/s43440-020-00155-6
|
| [3] |
R. Yan, Y. Zhang, Y. Li, L. Xia, Y. Guo, Q. Zhou, Structural basis for the recognition of SARS-CoV-2 by full-length human ACE2, Science, 367 (2020), 1444–1448. https://doi.org/10.1126/science.abb2762 doi: 10.1126/science.abb2762
|
| [4] | A. Elfiky, N. S. Ibrahim, Anti-SARS and anti-HCV drugs repurposing against the Papain-like protease of the newly emerged coronavirus (2019-nCoV), preprint. https://doi.org/10.21203/rs.2.23280/v1 |
| [5] | R. Arya, A. Das, V. Prashar, M. Kumar, Potential inhibitors against papain-like protease of novel coronavirus (COVID-19) from FDA approved drugs, preprint, chemrXiv. https://doi.org/10.26434/chemrxiv.11860011 |
| [6] |
D. Zhang, K. Wu, X. Zhang, S. Deng, B. Peng, In silico screening of Chinese herbal medicines with the potential to directly inhibit 2019 novel coronavirus, J. Integr. Med., 18 (2020), 152–158. https://doi.org/10.1016/j.joim.2020.02.005 doi: 10.1016/j.joim.2020.02.005
|
| [7] |
A. S. Hauser, M. M. Attwood, M. Rask-Andersen, H. B. Schiöth, D. E. Gloriam, Trends in GPCR drug discovery: new agents, targets and indications, Nat. Rev. Drug Discovery, 16 (2017), 829–842. https://doi.org/10.1038/nrd.2017.178 doi: 10.1038/nrd.2017.178
|
| [8] | N. Wang, Y. Li, S. Liu, L. Gao, C. Liu, X. Bao, et al., Bioinformatics analysis and validation of differentially expressed microRNAs with their target genes involved in GLP-1RA facilitated osteogenesis, Curr. Bioinf., 16 (2021), 928–942. https://dx.doi.org/10.2174/1574893615999200508091615 |
| [9] |
D. Weininger, SMILES: a chemical language and information system, J. Chem. Inf. Comput. Sci., 28 (1988), 31–36. https://doi.org/10.1021/ci00057a005 doi: 10.1021/ci00057a005
|
| [10] |
B. R. Beck, B. Shin, Y. Choi, S. Park, K. Kang, Predicting commercially available antiviral drugs that may act on the novel coronavirus (SARS-CoV-2) through a drug-target interaction deep learning model, Comput. Struct. Biotechnol. J., 18 (2020), 784–790. https://doi.org/10.1016/j.csbj.2020.03.025 doi: 10.1016/j.csbj.2020.03.025
|
| [11] | T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in Proceedings of the International Conference on Learning Representations (ICLR), (2017). https://arXiv.org/abs/1609.02907 |
| [12] |
H. Jiang, P. Cao, M. Xu, J. Yang, Z. Osmar, Hi-GCN: a hierarchical graph convolution network for graph embedding learning of brain network and brain disorders prediction, Comput. Biol. Med., 127 (2020), 104096. https://doi.org/10.1016/j.compbiomed.2020.104096. doi: 10.1016/j.compbiomed.2020.104096
|
| [13] | P. Veličković, G. Cucurull, A. Casanova, A. Romero, P Liò, Y. Bengio, Graph attention networks, in Proceedings of the International Conference on Learning Representations (ICLR), (2018). https://doi.org/10.17863/CAM.48429 |
| [14] |
C. Yang, P. Wang, J. Tan, Q. Liu, X. Li, Autism spectrum disorder diagnosis using graph attention network based on spatial-constrained sparse functional brain networks, Comput. Biol. Med., 139 (2021), 104963–104963. https://doi.org/10.1016/j.compbiomed.2021.104963 doi: 10.1016/j.compbiomed.2021.104963
|
| [15] | K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks?, in Proceedings of the International Conference on Learning Representations (ICLR), 2019. https://arXiv.org/abs/1810.00826 |
| [16] |
H. Öztürk, A. Özgür, E. Ozkirimli, DeepDTA: deep drug-target binding affinity prediction, Bioinformatics, 34 (2018), 821–829. https://doi.org/10.1093/bioinformatics/bty593 doi: 10.1093/bioinformatics/bty593
|
| [17] | H. Öztürk, E. Ozkirimli, A. Özgür, WideDTA: prediction of drug-target binding affinity, preprint, arXiv: 1902.04166. https://arXiv.org/abs/1902.04166v1 |
| [18] | X. Guo, W. Zhou, B. Shi, et al., An efficient multiple kernel support vector regression model for assessing dry weight of hemodialysis patients, Curr. Bioinf., 16 (2021), 284–293. https://dx.doi.org/10.2174/1574893615999200614172536 |
| [19] |
N. J. Reynolds, S. W. Mccombie, B. B. Shankar, W. R. Bishop, G. J. Fisher, SCH 47112, a novel staurosporine derivative, inhibits 12-O-tetradecanoylphorbol-13-acetate-induced inflammation and epidermal hyperplasia in hairless mouse skin, Arch. Dermatol. Res., 289 (19979), 540–546. https://doi.org/10.1007/s004030050236 doi: 10.1007/s004030050236
|
| [20] |
I. Kruman, Q. Guo, M. P. Mattson, Calcium and reactive oxygen species mediate staurosporine-induced mitochondrial dysfunction and apoptosis in PC12 cells, J. Neurosci. Res., 51 (1998), 293–308. https://doi.org/10.1002/(SICI)1097-4547(19980201)51:3<293::AID-JNR3>3.0.CO;2-B doi: 10.1002/(SICI)1097-4547(19980201)51:3<293::AID-JNR3>3.0.CO;2-B
|
| [21] |
Y. Mizushina, N. Maeda, I. Kuriyama, H. Yoshida, Dehydroaltenusin is a specific inhibitor of mammalian DNA polymerase α, Expert Opin. Invest. Drugs, 20 (2011), 1523. https://doi.org/10.1517/13543784.2011.619977 doi: 10.1517/13543784.2011.619977
|
| [22] |
E Fliszár-Nyúl, B. Lemli, S. Kunsági-Máté, L. Dellafiora, C. Dall'Asta, G. Cruciani, et al., Interaction of mycotoxin alternariol with serum albumin, Int. J. Mol. Sci., 20 (2019), 2352. https://doi.org/10.3390/ijms20092352 doi: 10.3390/ijms20092352
|
| [23] |
J. Duan, Y. Guan, F. Mu, C. Guo, E. Zhang, Y. Yin, et al., Protective effect of butin against ischemia/reperfusion-induced myocardial injury in diabetic mice: involvement of the AMPK/GSK-3β/Nrf2 signaling pathway, Sci. Rep., 7 (2017), 41491. https://doi.org/10.1038/srep41491 doi: 10.1038/srep41491
|
| [24] |
S. Howard, V. Berdini, J. A. Boulstridge, M. G. Carr, D. M. Cross, J. Curry, et al., Fragment-based discovery of the pyrazol-4-yl urea (AT9283), a multitargeted kinase inhibitor with potent aurora kinase activity, J. Med. Chem., 52 (2009), 379–388. https://doi.org/10.1021/jm800984v doi: 10.1021/jm800984v
|
| [25] |
H. Li, Y. Gong, Y. Liu, H. Lin, G. Wang, Detection of transcription factors binding to methylated DNA by deep recurrent neural network, Briefings Bioinf., 2021 (2021), https://doi.org/10.1093/bib/bbab533. doi: 10.1093/bib/bbab533
|
| [26] |
L. Zhang, Y. Yang, L. Chai, Q. Li, J. Liu, H. Lin, et al., A deep learning model to identify gene expression level using co-binding transcription factor signals, Briefings Bioinf., 2021 (2021), https://doi.org/10.1093/bib/bbab501 doi: 10.1093/bib/bbab501
|
| [27] |
H. Lv, L. Shi, J. W. Berkenpas, F. Y. Dao, H. Zulfiqar, H. Ding, et al., Application of artificial intelligence and machine learning for COVID-19 drug discovery and vaccine design, Briefings Bioinf., 22 (2021), 1–10. https://doi.org/10.1093/bib/bbab320 doi: 10.1093/bib/bbab320
|
| [28] | Z. Chen, Q. Zhou, A. Khan, J. Jill, R. Xiong, X. Liu, New trends of deep learning in clinical cardiology, Curr. Bioinf., 16 (2021), 954–962. https://dx.doi.org/10.2174/1574893615999200719234517 |
| 1. | Peng Li, Mosharaf Md Parvej, Chenghao Zhang, Shufang Guo, Jing Zhang, Advances in the Development of Representation Learning and Its Innovations against COVID-19, 2023, 3, 2673-8112, 1389, 10.3390/covid3090096 | |
| 2. | Seema Yadav, Abhishek Singh, Rishika Singhal, Jagat Pal Yadav, Revolutionizing drug discovery: The impact of artificial intelligence on advancements in pharmacology and the pharmaceutical industry, 2024, 2, 2949866X, 367, 10.1016/j.ipha.2024.02.009 |
Figures(3) / Tables(7)
Xianfang Wang, Qimeng Li, Yifeng Liu, Zhiyong Du, Ruixia Jin. Drug repositioning of COVID-19 based on mixed graph network and ion channel[J]. Mathematical Biosciences and Engineering, 2022, 19(4): 3269-3284. doi: 10.3934/mbe.2022151
| Applications | Refs. | Year | Multi-sensor | Methods |
| Neurologic Rehabilitation Trainings | [14] | 2018 | FSR, Wearable+force+embedded sensors | Rigid discs Electronic circuit |
| Physical Activity | [15] | 2021 | Kinect sensor ECG |
MOCAP TEL |
| Heart-Rate Monitor | [17] | 2020 | Wearable sensors | Polar Flow web |
| [18] | 2023 | Temperature+heart rate+acceleration+smoke sensors | Wireless Network | |
| Bio signals | [21] | 2018 | PPG EEG Body motion capture GSR Inertial sensors |
SSVEP |
| Vehicle Motion State Estimation | [23] | 2020 | ESC GNSS IMU+Series-grade chassis sensors |
Intrinsic Euler angles Immersion transformation |
| Human Identification and Activity | [32] | 2022 | PIR PRF-PIR IMU MI-PIR |
RGBD CNN NBC FMCW RNN |
| Infrastructure-free Localization in Vision-impaired Environments | [33] | 2010 | EO IMU RF |
HMD Kalman filter |
DownLoad:
CSV
| Device/Software | Version/Parameters |
| Inertial Sensor | Gyroscope: ±2000 dps, Accelerometer: ±16 g, Magnetometer: ±8.1 Ga |
| Flex Sensor | 10 (5 per hand) |
| Kinect V2 | Body skeleton tracking |
| EMG Data Collector | Biometrics |
| Communication Technology | Dual-band (2.4 GHz/5.8 GHz), Sync accuracy: 10 µs |
| ROS | Melodic Morenia (1.14.11) |
| Ubuntu | 20.04 LTS |
| MATLAB | R2023a |
| PICO 4 VR System | Latest firmware as of June 2023 |
| Unity | 2023.1.2 |
DownLoad:
CSV
| Algorithm | Accuracy | Precision | Recall | F1 Score | Average Prediction Time (ms) |
| GRU-RNN | 0.97 | 0.98 | 0.96 | 0.97 | 4.5 |
| Feedforward Neural Network | 0.85 | 0.87 | 0.83 | 0.85 | 5.2 |
| Support Vector Machine | 0.81 | 0.83 | 0.79 | 0.81 | 6.3 |
| Random Forest | 0.76 | 0.77 | 0.75 | 0.76 | 7.8 |
DownLoad:
CSV
| Applications | Refs. | Year | Multi-sensor | Methods |
| Neurologic Rehabilitation Trainings | [14] | 2018 | FSR, Wearable+force+embedded sensors | Rigid discs Electronic circuit |
| Physical Activity | [15] | 2021 | Kinect sensor ECG |
MOCAP TEL |
| Heart-Rate Monitor | [17] | 2020 | Wearable sensors | Polar Flow web |
| [18] | 2023 | Temperature+heart rate+acceleration+smoke sensors | Wireless Network | |
| Bio signals | [21] | 2018 | PPG EEG Body motion capture GSR Inertial sensors |
SSVEP |
| Vehicle Motion State Estimation | [23] | 2020 | ESC GNSS IMU+Series-grade chassis sensors |
Intrinsic Euler angles Immersion transformation |
| Human Identification and Activity | [32] | 2022 | PIR PRF-PIR IMU MI-PIR |
RGBD CNN NBC FMCW RNN |
| Infrastructure-free Localization in Vision-impaired Environments | [33] | 2010 | EO IMU RF |
HMD Kalman filter |
| Device/Software | Version/Parameters |
| Inertial Sensor | Gyroscope: ±2000 dps, Accelerometer: ±16 g, Magnetometer: ±8.1 Ga |
| Flex Sensor | 10 (5 per hand) |
| Kinect V2 | Body skeleton tracking |
| EMG Data Collector | Biometrics |
| Communication Technology | Dual-band (2.4 GHz/5.8 GHz), Sync accuracy: 10 µs |
| ROS | Melodic Morenia (1.14.11) |
| Ubuntu | 20.04 LTS |
| MATLAB | R2023a |
| PICO 4 VR System | Latest firmware as of June 2023 |
| Unity | 2023.1.2 |
| Algorithm | Accuracy | Precision | Recall | F1 Score | Average Prediction Time (ms) |
| GRU-RNN | 0.97 | 0.98 | 0.96 | 0.97 | 4.5 |
| Feedforward Neural Network | 0.85 | 0.87 | 0.83 | 0.85 | 5.2 |
| Support Vector Machine | 0.81 | 0.83 | 0.79 | 0.81 | 6.3 |
| Random Forest | 0.76 | 0.77 | 0.75 | 0.76 | 7.8 |
