Figure 1.
Structure of an LSTM unit.
Citation: Marie K. Fialkowski, Ashley Yamanaka, Lynne R. Wilkens, Kathryn L. Braun, Jean Butel, Reynolette Ettienne, Katalina McGlone, Shelley Remengesau, Julianne M. Power, Emihner Johnson, Daisy Gilmatam, Travis Fleming, Mark Acosta, Tayna Belyeu-Camacho, Moria Shomour, Cecilia Sigrah, Claudio Nigg, Rachel Novotny. Recruitment Strategies and Lessons Learned from the Children’s Healthy Living Program Prevalence Survey[J]. AIMS Public Health, 2016, 3(1): 140-157. doi: 10.3934/publichealth.2016.1.140
| [1] | Wenbo Yang, Wei Liu, Qun Gao . Prediction of dissolved oxygen concentration in aquaculture based on attention mechanism and combined neural network. Mathematical Biosciences and Engineering, 2023, 20(1): 998-1017. doi: 10.3934/mbe.2023046 |
| [2] | Chongyi Tian, Longlong Lin, Yi Yan, Ruiqi Wang, Fan Wang, Qingqing Chi . Photovoltaic power prediction based on dilated causal convolutional network and stacked LSTM. Mathematical Biosciences and Engineering, 2024, 21(1): 1167-1185. doi: 10.3934/mbe.2024049 |
| [3] | Mohanad A. Deif, Ahmed A. A. Solyman, Mehrdad Ahmadi Kamarposhti, Shahab S. Band, Rania E. Hammam . A deep bidirectional recurrent neural network for identification of SARS-CoV-2 from viral genome sequences. Mathematical Biosciences and Engineering, 2021, 18(6): 8933-8950. doi: 10.3934/mbe.2021440 |
| [4] | Xiaotong Ji, Dan Liu, Ping Xiong . Multi-model fusion short-term power load forecasting based on improved WOA optimization. Mathematical Biosciences and Engineering, 2022, 19(12): 13399-13420. doi: 10.3934/mbe.2022627 |
| [5] | Hui Li, Xintang Liu, Dongbao Jia, Yanyan Chen, Pengfei Hou, Haining Li . Research on chest radiography recognition model based on deep learning. Mathematical Biosciences and Engineering, 2022, 19(11): 11768-11781. doi: 10.3934/mbe.2022548 |
| [6] | Zhangjie Wu, Minming Gu . A novel attention-guided ECA-CNN architecture for sEMG-based gait classification. Mathematical Biosciences and Engineering, 2023, 20(4): 7140-7153. doi: 10.3934/mbe.2023308 |
| [7] | Sakorn Mekruksavanich, Wikanda Phaphan, Anuchit Jitpattanakul . Epileptic seizure detection in EEG signals via an enhanced hybrid CNN with an integrated attention mechanism. Mathematical Biosciences and Engineering, 2025, 22(1): 73-105. doi: 10.3934/mbe.2025004 |
| [8] | Liguo Zhang, Liangyu Zhao, Yongtao Yan . A hybrid neural network-based intelligent body posture estimation system in sports scenes. Mathematical Biosciences and Engineering, 2024, 21(1): 1017-1037. doi: 10.3934/mbe.2024042 |
| [9] | Jing Wang, Jiaohua Qin, Xuyu Xiang, Yun Tan, Nan Pan . CAPTCHA recognition based on deep convolutional neural network. Mathematical Biosciences and Engineering, 2019, 16(5): 5851-5861. doi: 10.3934/mbe.2019292 |
| [10] | Tao Yang, Qicheng Yang, Yibo Zhou, Chuanbiao Wen . Glucose trend prediction model based on improved wavelet transform and gated recurrent unit. Mathematical Biosciences and Engineering, 2023, 20(9): 17037-17056. doi: 10.3934/mbe.2023760 |
Handwriting is a complex and essential communication skill for humans that is often disregarded. Nonetheless, handwriting (both manuscript and cursive) is a critical motor skill for children's cognitive development and academic success [1]. Considering as an example the case of a musical composition, experts in this area show a preference for handwriting editions over the use of specialized, user-friendly interfaces. The latter due to the lack of confidence in using a computer for artistic applications. Despite this fact, the automatic detection of hand-written musical notation in digital media is highly recommended and profitable in such an application, similar to characters recognition in text analysis. Calvo-Zaragoza and Oncina proposed a music notation recognition method based on an electronic pen and a finite-state machine to detect user strokes competitively [2].
In hand gesture analysis, Mendes et al. studied the interaction human and robot based on gesture recognition by developing an Electromyography sensor to capture and discriminate valid gestures using a Deep Learning technique [3].
Analogously, Słapekand and Paskiel proposed a method based on Bèzier curves to extract predefined hand gestures without using a marker at the start or end in the sequence of points [4]. Such type of methods tries to detect the handwriting style to develop improved methodologies in smart human-computer interfaces and handwriting recognition.
On the other hand, the development of brain-computer interfaces offers an alternative for clinical and assisted applications involving motor biosignals. For instance, the communication abilities of people with or without disabilities can be enhanced through devices interfaced with Electroencephalography (EEG) and/or Electromyography (EMG) signals.
There are two kinds of EMG [5]: Surface EMG and intramuscular EMG. Intramuscular EMG is a technique that is performed by inserting electrodes into a skeletal muscle. In contrast, surface EMG is performed by placing electrodes on the skin surface above a muscle. Compared to intramuscular EMG, surface EMG has the advantage of being a non invasive technique which is clinically superior in the diagnosis, rehabilitation, and testing of new therapies linked to motor activity. Surface EMG (sEMG) signals form an intricate interference pattern of bioelectrical signals generated mainly from the dynamical activity of skeletal muscles. As a consequence, sEMG signals from individual muscles cannot be reliably distinguished.
Additionally, sEMG has found several new applications supported by Machine Learning and Deep Learning methods as movement analysis, muscular fatigue, sports therapies, wearable devices, and particularly in handwriting recognition [6]. sEMG data acquisition and the corresponding waveforms are affected by many factors such as the muscles' physiological properties, added noise in the instrumentation used for sensing, as well as the chosen location of the electrodes related to the active muscle units. Hence, conventional features extraction methods involving sEMG generally require robust pre-processing steps and filtering. Moreover, the sEMG signal complexity can also be explained by the fact that there are various ways in which a motor task can be performed (motor redundancy) [7,8]. Simao et al. made a recent review of the sEMG-based systems using pattern recognition techniques and describing sensitive processes as signal acquisition and filtering. Their study focused on new human-machine interaction modalities [9].
Recently, advances in biomedicine have made possible the invention of myoelectric prostheses capable of performing real arm or leg movements [10]. Neural prostheses strive to restore the mobility of limbs, and the ability to communicate of subjects with disabilities, through the interconnection of brain potentials, actuators, or artificial devices in functional electrical stimulation. A significant innovation was the implementation of a device to predict a premature birth several weeks in advance. In this context, bioelectric interfaces allow extracting handwriting patterns for diagnostic purposes in patients with Parkinson's disease, and more recently, dysgraphia disability. Therefore, sEMG signals generated during the handwriting can be used to characterize neurological disorders [11]. In handwriting, the sEMG signals are generated and captured from the hand and forearm muscles during the handwriting activity, to be decoded by the specific purpose recognition systems [12]. In early studies, Linderman et al. developed a reliable system using sEMG signals to recognize handwriting characters to diagnose or measure disease progression in medical treatments [13].
In general, handwriting recognition from sEMG signals is addressed using the following approaches: Template Matching [13], Dynamic Time Warping (DTW) [12,14], Radial Basis Functions (RBF) Neural Networks [11]. It is noteworthy that sEMG signals convey a great amount of information about limb movements. Following this reasoning, several applications have been proposed in the literature for the recognition, modeling, and reconstruction of handwriting movements. Additionally, the reconstruction of handwriting from sEMG signals has been addressed using the Wiener filters [13], Recursive Least Squares [15], RBF neural networks [16], and Kalman filters [17]. Therefore, these advances are not only limited to medical purposes, but also toys and video games can be controlled through the musculature’s bioelectric activity. In a related work, Wei et al. devised a gesture recognition method by using multi-view features for both unimodal and multimodal sEMG-based gestures driven by a Deep convolutional neural network (DCNN) model [18].
Dash et al. [19] designed an approach called AirScript, for visualizing and recognizing characters written in the air. In this case, IMU (Inertial Measurement Unit) data from the Myo armband were applied to obtain a 2D coordinates sequence to visualize a handwritten digit as an image. For handwritten digit recognition, a convolutional neural network (CNN), and two GRU (Gated Recurrent Unit) neural networks were analyzed. The images were used as inputs to the CNN, the generated sequence of 2D coordinates was utilized as input to the first GRU neural network, and the raw IMU signal as input to the second GRU neural network. Subsequently, the outputs of those three neural networks were fused to yield the class of the recognized digit. The system reached an accuracy of 91.7 and 96.7% for independent and dependent tasks, respectively. In the method implemented by Roy et al. [20], single stroke digits were written in the air using a marker of uniform color. A video camera recorded the handwriting trajectory. After segmenting each video frame, the marker tip position was used to approximate the handwriting trajectory and to obtain an image of the handwritten digit. Next, a CNN was used for recognizing the image of each handwritten digit reporting an accuracy of 97.7, 95.4 and 93.7% for the English, Bengali, and Devanagari digits, respectively.
In some studies, the recognition of handwriting characters was limited to use small databases, imposing several restrictions on the placement of the electrodes, and in the number and complexity of handwriting strokes. Most handwriting recognition methods using sEMG signals were mainly based on pre-processing and feature extraction [13], while some others use the DTW method after pre-processing [14].
The approach discussed in this paper addreses handwriting character recognition problem from a more general point of view. To cope with the variability of the sEMG signals, the use of CNNs is suggested for the signal processing and feature extraction [21,22]. Hence, a Recurrent Neural Network (RNN) based on the Long Short-Term Memory (LSTM) and GRU networks is used to recognize sEMG signals, considering that LSTMs have also been previously applied successfully to gesture recognition [23]. Implemented CNN architectures are based on several studies demonstrating the efficacy and robustness of CNNs for extracting discriminant features directly from raw data. CNNs use a multi-level representation to learn and decode original data from specialized layers applied to an optimal recognition task [24].
A convolutional neural network (CNN), is a well-known Deep Learning architecture originally inspired by findings in visual perception. In 1959, Hubel & Wiesel discovered the existence of simple cells in the visual cortex of cats [25]. Simple cells are neurons that respond to bars or edges directed to a specific orientation. Hubel & Wiesel also recommended a hierarchical model to explain the behavior of simple cells. According to Marr [26], the receptive fields of simple cells are either bar o edge-shaped, and they compute a linear convolution with a bar or edge-shaped mask. Inspired on these findings, Fukushima suggested the Neocognitron in 1980 [27], which perhaps constitutes one of the first CNN architectures. The Neocognitron introduced two basic layer types: Convolutional layer and subsampling layer. Weng et al. proposed the Cresceptron architecture, which introduced max-pooling [28] and a variant called blurring [29].
In 1990, LeCun et al. created the LeNet architecture to classify handwritten digits [30]. The LeNet architecture and variants were the first to be trained with the backpropagation algorithm [31]. This architecture obtained effective representations directly from the image, which made it possible to recognize visual patterns from raw pixels with little-to-none pre-processing. At the same time, Zhang et al. used a Shift-Invariant Artificial Neural Network (SIANN) to recognize characters in an image [32].
In the beginning, despite the success of the LeNet-5 for digit recognition, CNNs were not successful for more complex tasks. The main reasons included the low computing power available, the small number of training samples, and diverse problems that arose especially for Deep CNNs (DCNN) [33]. After several efforts to surmount the difficulties for training DCNNs and thanks to the availability of new powerful computing resources, Krizhevsky et al. presented the AlexNet architecture, which won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC2012) [34]. The overall AlexNet architecture [35] is basically the same as the LeNet-5 architecture but considerably deeper, with stacked convolutions and with more filters per layer. Among the innovations incorporated in AlexNet, were the use of the Rectified Linear Unit (ReLU) activation functions [36], max-pooling layers, dropout [37], data augmentation [38], and ad-hoc initialization algorithms, Jiuxiang Gu et al. detail the recent improvements of CNNs related to the aforementioned innovations [33].
Several years later, He et al. proposed the ResNet architecture, which won the ILSVRC 2015 [39]. It was several times deeper than AlexNet and incorporated innovations such as residual blocks, and batch-normalization layers [40].
Recurrent neural networks are a class of neural networks originally conceived as en effort to incorporate time in neural networks. In 1986 Jordan et al. [41] proposed a neural network with dynamic memory which incorporated recurrent connections from the output layer to the hidden layer. Several years later, Elman et al. [42] proposed another recurrent network based on the Jordan network. This model incorporated recurrent connections directly from the output of the hidden layer allowing the network to maintain a sort of state or memory in the hidden layer. This allowed the network to perform difficult tasks such as temporal sequence prediction, sequence-to-sequence mapping, etc.
Recurrent neural networks are trained by using Backpropagation-Through-Time (BPTT). First, the network is unfolded in time. As a result, a very deep shared weights feedforward neural network is obtained. Then training is performed by using the back-propagation algorithm. At that time, recurrent neural networks had a limited success because they presented learning problems when dealing with very long sequences [43,44]. This was called the vanishing gradient problem because the backpropagated gradients used to update the weights vanished rapidly to zero. As a result, the network was not able to learn patterns contained at the beginning of the sequence. This means that the network was not able to learn long-range dependencies.
In 1997 Hochreiter & Schmidhuber [45] proposed Long Short-Term Memory (LSTM) networks as solution to the vanishing gradient problem. The basic structure of an LSTM network is composed of a chain of units that can decide when to remember, forget, and acquire new information. Consequently, an LSTM unit is composed of one memory cell and three multiplicative gates: Forget, input, and output gates. Figure 1 illustrates the basic structure of an LSTM unit, including in essence sigmoid and tanh units, adders, and multipliers.
Mathematically, the equations governing the operation of an LSTM unit are described next.
The forget gate output is given by Eq (2.1):
|
$ $
|
(2.1) |
where $ \sigma(x) = \frac{\mathrm{1}}{\mathrm{1} + e^{-x}} \, \in [0, 1] $ is the sigmoid activation function, $ x_t $ is the current input, $ h_t $ is the current output of the LSTM unit, $ W $ and $ b $ are weight and bias matrices, respectively.
The output of the input gate $ i_t $ is given by Eq (2.2):
|
$ $
|
(2.2) |
where $ h_{t-1} $ is the previous output, $ W $ and $ b $ are respectively, the weight matrix and bias parameters that must be learned. Besides, the current output of the cell state $ c_t $, is obtained by Eq (2.3):
|
$ $
|
(2.3) |
where $ tanh(x) $ is the hyperbolic tangent activation function. Finally, $ o_t $ is the output of the output gate defined as Eq (2.4):
|
$ $
|
(2.4) |
LSTM networks have found countless applications since their invention, which involve speech recognition, text analysis, language processing, machine translation, handwriting generation, among others.
Gated Recurrent Unit (GRU) networks were introduced by Cho [46] as an alternative to LSTM networks with fewer parameters. A GRU unit utilizes only two gates to decide when to acquire new information and its hidden state is also its output. Basically, a GRU unit is composed of one memory cell and two multiplicative gates: Update, and output gates. Figure 2 illustrates the structure of a fundamental GRU unit.
The computations performed by a GRU unit are described next. The reset gate's output $ r_t $, which decides the amount of information to forget, is defined as Eq (2.5)
|
$ $
|
(2.5) |
where $ \sigma $ is a sigmoid activation function, $ x_t $ is the current input, $ h_{t-1} $ is the hidden state of the previous time step, $ W $ and $ b $ are the weight matrix and bias parameters, respectively. The output of the update gate for time step $ t $, $ z_t $, is described by Eq (2.6)
|
$ $
|
(2.6) |
Finally, the current hidden state (or output) $ h_t $ of the GRU unit is formally given by Eq (2.7)
|
$ $
|
(2.7) |
where $ \tilde{h}_t $ is simply defined as,
|
$ $
|
(2.8) |
which uses the reset gate output to feedback past relevant information.
Most previous approaches to handwriting character recognition from sEMG signals were based on pre-processing and feature extraction followed by a linear classifier such as LDA [13]. Whereas some other approaches were based on pre-processing followed by the DTW method. Nevertheless, all these approaches are highly restrictive, in the sense that they require the electrodes to be carefully placed on the muscles to facilitate classification. Additionally, all those approaches were only designed for single-stroke characters.
In this paper, a more general and reliable approach is proposed. This method disposes of highlighted features as multi-user, free-style, multi-stroke handwriting character recognition. Each test-subject wears a Myo armband device on the forearm, and no restriction is made regarding a predefined position of the electrodes. Figure 3 shows the global components of the database construction of the characters recognition approach.
The main contributions are focused on the database creation and the development of a discriminant and versatile characters recognition system. The four blocks are dedicated to the character's acquisition system and especially to the construction of the database using software programmed to control the captured patterns (see details in Section 3). To cope with the variability of sEMG signals, powerful techniques for feature extraction, and sequence processing, such as CNNs and RNNs (LSTM, GRU) networks, were respectively used. From a multi-channel sEMG signal approach, four neural network architectures were designed and implemented, which were named DCNN-LSTM, DCNN-2LSTM, DCNN-GRU, and DCNN-2GRU architectures.
Figure 4 shows the internal layers of the proposed CNN-LSTM architecture.
Such a DCNN-LSTM architecture comprises two convolutional layers, one pooling layer, one LSTM layer, and one fully connected layer. Each convolutional layer has a ReLU activation function and 32 filters of size $ 3 \times 1 $. A max-pooling layer follows the first convolutional layer. Thereunto, a dropout layer is added to avoid over-training and improve generalization in each convolutional layer. Besides, the LSTM layer is composed of 180 units. Conversely, the fully connected layer has 36 outputs and a SoftMax activation function.
The neural architectures were implemented in Python using Keras with Tensorflow backend with: The loss function based on categorical-cross-entropy, the optimizer known as Nadam, the fundamental metrics defined as loss and accuracy.
Apart from the above, the internal layers of the proposed CNN-GRU architecture are shown in Figure 5.
It is worth mentioning that the DCNN-GRU architecture is similar to the DCNN-LSTM architecture. The main difference relies on using a GRU layer instead of an LSTM layer. Whereas, the DCNN-2LSTM and DCNN-2GRU architectures use two stacked LSTM and GRU layers, respectively.
Figure 6 presents the sequence of processes involved in the identification of the character class.
The most important steps are armband calibration, handwriting task, sEMG capture (Myo armband: Raw sEMG signals generation, filtering and sampling), magnitude and sequence length normalization to finally send the 8-channel sEMG data to the neural architecture for recognition. In concrete terms, the input to the Neural Network is an array of size $ (900, 8) $ with eight-channel sEMG signals, one pattern obtained from the armband at each capture, the number of outputs are 36. Each output represents an estimation of a class probability.
The raw sEMG data used in the numerical tests were collected from a MyoTM armband. The MyoTM armband is a commercial electronic device developed by Thalmic LabsⓇ. Such a system is a low-cost, non-invasive, user-friendly wireless eight-channel device with dry electrodes working at a sampling frequency of 200 Hz.
A capture application was developed in Python, under Windows 10. Synthetically, the capture procedure operates as follows, first, the application to capture sEMG signals is started from a command prompt. The application displays a GUI with an option of terminating the application at any time. Next, a subject wears a Myo armband on the left or right forearm (see Figure 7). Posteriorly, the armband is turned on and connected to the PC via a Bluetooth dongle that is plugged into a USB port. Once the device has been correctly detected, a personal calibration profile must be created and selected by using the Myo armband Manager software. Then, the user types a character (letter or digit) into a text field in the GUI. If the typed character (or class) is wrongly chosen, the software still provides the possibility to cancel the capture. After that, the user clicks on Start button to begin the capture process. At this time the subject starts writing with the index finger (or a stylus pen) on the touchscreen. The capture process finishes when the subject clicks on Stop button. Now, the subject must click on Save button to write the captured data to a CSV file. The captured data comprises eight signals corresponding to the eight sensors of the Myo armband. Such characters are written in a freehand style with the index finger or a stylus pen. Additionally, the (x, y) coordinates of the handwriting movements are visualized on the screen. Thereunto, remember that before each handwriting task, the Myo armband is calibrated so that the device always started with the same personal configuration for different users.
Three healthy subjects seated in a comfortable position participated in the creation of the database.
Each task consisted of drawing with the index finger (or stylus pen), 26 lowercase characters ('a'-'z'), and 10 decimal digits ('0'-'9'). The subjects were free to use their particular handwriting style with one or more strokes.
Because a total of 3 subjects participated in creating the database, the number of classes of characters is 36 (26 letters and 10 digits), and each subject wrote each character class one thousand times, the total number of patterns is 108,000 (i.e., $ 3\times36\times 1000 $). Ergo, the database contains 108,000 patterns of multi-channel sEMG signals divided into 36 classes and 3,000 patterns per class. Additionally, the database also provides the trajectory of each handwritten character along with the stroke number. This can be useful for visualization or research purposes (i.e., handwriting trajectory reconstruction). Figure 9 shows the representative (average) pattern over all classes. The amplitudes are small in the right side because there are fewer patterns with lengths above 2.5 seconds.
The database was split into training and testing sets to assess the performance of the proposed architectures. The testing set contains 10% of the samples, whereas the rest of the samples formed the training set. Thus, 97,200 samples were used for training and 10,800 samples were used for testing.
All architectures were trained and tested on a workstation with an IntelⓇ CoreTM i7 processor, 16 GB RAM, an NvidiaⓇ GeForce RTXTM 2080 GPU, and Windows 10 using Keras with a TensorflowTM backend. For all cases, the Nesterov-accelerated ADAptive Moment (NADAM) algorithm [47] was used for training using the cross-entropy loss function and a batch size of 1024.
Firstly, each architecture was trained with similar tuning parameters for 8,000 epochs using a cyclical learning rate schedule [48]. Thus, the minimum and maximum learning rates were set to 0.000001 and 0.001, respectively. Furthermore, the step size was defined to five times the number of iterations in each epoch. Secondly, each training essay was repeated several times with different initial conditions to help avoid local minima. Moreover, the dropout rate was varied from 0.8 to 0.6 to help reduce the effects of overfitting. Likewise, the number of units in the LSTM and GRU layers varied from 80 to 200.
Table 1 shows the results obtained for the four architectures using only samples for one subject.
| Architecture | Accuracy (%) |
| CNN-LSTM | 96.37 |
| CNN-2LSTM | 96.48 |
| CNN-GRU | 97.03 |
| CNN-2GRU | 96.29 |
DownLoad:
CSV
As can be seen, the best accuracy (97.03%) is achieved by the CNN-GRU architecture, whereas the CNN-2GRU architecture achieves the lowest accuracy (96.29%).
Table 2 shows the results for the CNN-GRU architecture for each and all subjects.
| Subjects | Accuracy (%) |
| Subject1 | 97.04 |
| Subject2 | 95.05 |
| Subject3 | 94.89 |
| All | 94.85 |
DownLoad:
CSV
Note that the highest accuracy was obtained for subject1 and the lowest for subject3. The accuracy of all subjects was 94.85%, which is slower than the accuracy of a single subject. This effect is due to the higher variability of the sEMG signals of all subjects. Table 3 shows a quantitative comparison with representative state of the art studies.
| Method | Samples | Subjects | Classes | Accuracy (%) |
| Template Matching [13] | 350 | 6 | 10 | 97 |
| DTW [12] | 520 | 3 | 26 | 84.29 |
| DTW [14] | 780 | 4 | 26 | 78.24 to 92.42 |
| Proposed | 108k | 3 | 30 | 94.85 |
DownLoad:
CSV
From the numerical results, it can be observed and concluded that the proposed approach is highly competitive, and in general, compares well with the already-existing methods in the state of the art. Figure 10 shows the confusion matrix obtained for the CNN-GRU architecture.
It is noteworthy that all the classes are recognized with high accuracy for samples from all test subjects. Figure 11 shows the ROC curve obtained for class a by using the CNN-GRU architecture.
As can be seen, the ROC area is very close to one, which indicates an excellent separation capability between classes. A similar result was obtained for the other 35 classes.
The numerical results showed that the four architectures proposed presented comparable accuracies. However, the fastest architecture (CNN-GRU) also presented the highest average accuracy. It is worth mentioning that these results yielded a high accuracy, although no pre-processing was performed. For these reasons, the CNN-GRU architecture was chosen for the task of multi-user, free-style, handwriting character recognition. Notwithstanding, there are always options for amending the actual systems. For instance, an opportunity of improvement is represented by the possibility of testing our approach using a more extensive multi-user database of sEMG signals.
A new methodology for recognizing multi-stroke handwriting characters from sEMG signals using an MyoTM armband device has been presented. The proposed approach consists of using robust Deep Learning architectures for feature extraction, and sequence recognition algorithms such as CNNs and RNNs. One advantage of the discussed approach is that characters can be written either by using a pen or a finger. Another advantage is that characters can also be written in the air.
A CNN-GRU architecture achieved average accuracy of 94.85% for the task of multi-user handwriting character recognition. In comparison with previous works, the results obtained in this work are promising even though sEMG signals presented complex variations on their shapes due to the differences in the placement of the armband device. Therefore, this study is an essential contribution to the area of brain-computer interfaces. Brain-computer interfaces offer an alternative to restore or enhance the communications skills of people with and without disabilities.
This project was fully supported by the Electronics Department of the Universidad de Guanajuato under the Program POA 2020. Especially, authors would like to give special thanks to Perla Villegas-Torres for her support in the edition of this manuscript. Authors are also grateful to the Mexican Council of Science and Technology (CONACyT), for granting the scholarships (CVU/SGN: 495556/455203 and 495754/284009) to fund their doctoral studies.
The authors declare that there is no conflict of interests regarding the publication of this paper.
| [1] | Ng M, Fleming T, Robinson M, et al. (2014) Global, regional, and national prevalence of overweight and obesity in children and adults during 1980-2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet 384(9945): 766-81. |
| [2] | Hawley NL and McGarvey ST. (2015) Obesity and diabetes in Pacific Islanders: the current burden and the need for urgent action. Curr Diab Rep 15(5): 29. |
| [3] | McLennan AK and Ulijaszek SJ. (2015) Obesity emergence in the Pacific islands: why understanding colonial history and social change is important. Public health nutrition 18(08): 1499-1505. |
| [4] | Pacific Islands Health Officers Association, Board Resolution #48-01: The burden of non-communicable diseases. 2010, Pacific Islands Health Officers Association: Honolulu, HI. |
| [5] | Must A. (2003) Does overweight in childhood have an impact on adult health? Nutr Rev 61(4): 139-42. |
| [6] | Berenson GS, Srinivasan SR, Bao W, et al. (1998) Association between multiple cardiovascular risk factors and atherosclerosis in children and young adults. The Bogulasa Heart Study. N Engl J Med 338(23): 1650-1656. |
| [7] | Lauer RM and Clarke WR. (1989) Childhood risk factors for high adult blood pressure: the Muscatine Study. Pediatrics 84(4): 633-641. |
| [8] | Lauer RM, Lee J, and Clarke WR. (1988) Factors affecting the relationship between childhood and adult cholesterol levels: the Muscatine Study. Pediatrics 82(3): 309-318. |
| [9] | Association AD. (2000) Type 2 Diabetes in Children and Adolescents. Pediatrics 105(3): 671-680. |
| [10] | Novotny R. (2006) Nutrition and health status of children in the CNMI, in Preliminary Report: June 24, 2006. University of Hawaii, Northern Marianas College, Department of Public Health. |
| [11] | Novotny R, Coleman P, Tenorio L, et al. (2007) Breastfeeding is associated with lower body mass index among children of the Commonwealth of the Northern Mariana Islands. J Am Diet Assoc 107(10): 1743-6. |
| [12] | Guerrero RTL, Paulino YC, Novotny R, et al. (2008) Diet and obesity among Chamorro and Filipino adults on Guam. Asia Pac J Clin Nutr 17(2): 216-22. |
| [13] | Novotny R, Vijayadeva V, Grove J, et al. (2012) Dietary intake among Native Hawaiian, Filipino, and White children and caregivers in Hawai'i. Hawaii J Med Public Health 71(12): 353-8. |
| [14] | Derrickson J, Tanaka D, and Novotny R. (1997) Heights and weights of Head Start preschool children in Hawaii. J Am Diet Assoc 97(12): 1424-1426. |
| [15] | Paulino YC, Coleman P, Davison NH, et al. (2008) Nutritional characteristics and body mass index of children in the Commonwealth of the Northern Mariana Islands. J Am Diet Assoc 108(12): 2100-2104. |
| [16] | Novotny R, Williams AE, Vinoya AC, et al. (2009) US acculturation, food intake, and obesity among Asian-Pacific hotel workers. J Am Diet Assoc 109(10): 1712-1718. |
| [17] | Novotny R, Chen C, Williams AE, et al. (2012) US acculturation is associated with health behaviors and obesity, but not their change, with a hotel-based intervention among Asian-Pacific Islanders. J Acad Nutr Diet 112(5): 649-656. |
| [18] | Nigg C, Maddock J, Yamauchi J et al. (2005) The healthy Hawaii initiative: A social ecological approach promoting healthy communities. Am J Health Promot 19(4): 310-313. |
| [19] | Iversen CS, Nigg C, and Titchenal CA. (2011) The impact of an elementary after-school nutrition and physical activity program on children's fruit and vegetable intake, physical activity, and body mass index: Fun 5. Hawaii Med J 70(7 Suppl 1): 37-41. |
| [20] | Novotny R, Fialkowski MK, Li F, et al. (2015) Systematic review of prevalence of young child overweight and obesity in the United States-Affiliated Pacific region compared with the 48 contiguous states: The Children's Healthy Living Program. Am J Public Health 105(1): e22-e35. |
| [21] | Murphy SP. (2003) Collection and analysis of intake data from the integrated survey. J Nutr 133(2): p. 585S-589S. |
| [22] | Peterkin BB. (1981) Nationwide food consumption survey, 1977-1978. Prog Clin Biol Res 67: 59-69. |
| [23] | U.S. Central Intelligence Agency. The World Factbook. 2015. Available from: https://www.cia.gov/library/publications/the-world-factbook/. |
| [24] | U.S. Census Bureau, United States Census 2010. 2010: Jeffersonville, IN. |
| [25] | Central Intelligence Agency. Palau. World Fact Book, 2014. Available from: https://www.cia.gov/library/publications/the-world-factbook/geos/ps.html. |
| [26] | Novotny R, Areta A, Bersamin A, et al. (2013) The Pacific way to child wellness: the Children’s Healthy Living Program for remote underserved minority populations of the Pacific region (CHL). Hawaii J Med Public Health 72(11): 406-408. |
| [27] | Wilken LR, Novotny R, Fialkowski MK, et al. (2013) Children's healthy living (CHL) program for remote underserved minority populations in the pacific region: rationale and design of a community randomized trial to prevent early childhood obesity. BMC Public Health 13(1): 944. |
| [28] | Hooks PC, Tsong Y, Baranowski T, et al. (1988) Recruitment strategies for multiethnic family and community health research. Family and Community Health 11(1): 48-59. |
| [29] | Loxton D, Powers J, Anderson AE, et al. (2015) Online and Offline Recruitment of Young Women for a Longitudinal Health Survey: Findings From the Australian Longitudinal Study on Women's Health 1989-95 Cohort. J Med Internet Res 17(5): e109. |
| [30] | Markert J, Alff F, Zschaler S, et al. (2013) Prevention of childhood obesity: recruiting strategies via local paediatricians and study protocol for a telephone-based counselling programme. Obes Res Clin Pract 7(6): e476-e486. |
| [31] | Greany M, Lees FD, Nigg C, et al. (2006) Recruiting and Retaining Older Adults for Health Promotion Research: The Experience of the SENIOR Project. Journal of Nutrition for the Elderly 25(3-4): 3-22. |
| [32] | Dilworth-Anderson P, Thaker S, and Burke JM. (2005) Recruitment strategies for studying dementia in later life among diverse cultural groups. Alzheimer Dis Assoc Disord 19(4): 256-260. |
| [33] | Levkoff S and Sanchez H. (2003) Lessons learned about minority recruitment and retention from the Centers on Minority Aging and Health Promotion. Gerontologist 43(1): 18-26. |
| [34] | Saunders SD, Greaney ML, Lees FD, et al. Achieving recruitment goals through community partnerships: the SENIOR Project. Fam Community Health 26(3): 194-202. |
| [35] | Warren-Findlow J, Prohaska TR, and Freedman D. (2003) Challenges and opportunities in recruiting and retaining underrepresented populations into health promotion research. Gerontologist 43 Spec No 1: 37-46. |
| [36] | Luebbert R and Perez A. (2015) Barriers to Clinical Research Participation Among African Americans. J Transcult Nurs 1043659615575578. |
| [37] | Brown SD, Partee PN, Feng J, et al. (2015) Outreach to diversify clinical trial participation: A randomized recruitment study. Clin Trials 12(3): 205-211. |
| [38] | Recruitment for clinical trials: the Lipid Research Clinics Coronary Primary Prevention Trial experience. Its implications for future trials. Circulation, 1982. 66(6 Pt 2): IV1-78. |
| [39] | Corbie-Smith G, Thomas SB, and St George DM. (2002) Distrust, race, and research. Arch Intern Med 162(21): 2458-2463. |
| [40] | Roberson NL. (1994) Clinical trial participation. Viewpoints from racial/ethnic groups. Cancer 74(9 Suppl): 2687-2691. |
| [41] | Heller C, Balls-Berry JE, Nery JD, et al. (2014) Strategies addressing barriers to clinical trial enrollment of underrepresented populations: a systematic review. Contemp Clin Trials 39(2): 169-182. |
| [42] | Wasem C. (2004) U.S. affiliated Pacific basin jurisdictions: Legal, geographic, and demographic information. Office of Pacific Health & Human Services - Region IX Office of the Regional Health Administrator, HHS: Honolulu, HI. |
| [43] | Li F, Wilkens L, Novotny R, et al. (2014) Anthropometric standardization in the U.S. Affiliated Pacific: The Children’S Healthy Living Program (1024.6). The FASEB Journal 28(1 Supplement). |
| [44] | Novotny R, Li F, Fialkowski M, et al. (2015) Child Obesity and Acanthosis Nigricans in the Children's Healthy Living Program (CHL). The FASEB Journal 29(1 Supplement): 902-909. |
| [45] | Boushey C, Yonemori K, Novotny R, et al. (2015) Intakes of key dietary indicators among children 2-8 years participating in the Children's Healthy Living (CHL) Program. FASEB J 29(1 Supplement): 381-1. |
| [46] | Ettienne R, Boushey C, Yonemori K, et al. (2015) Young children's physical activities (PA) in the US Affiliated Pacific (USAP): The Children's Healthy Living (CHL) Program. The FASEB Journal 29(1 Supplement): 732-10. |
| [47] | Li F, Nigg C, McGlone K, et al. (2015) Young children's screen time and obesity in the US Affiliated Pacific: The Children's Healthy Living Program. The FASEB Journal 29(1 Supplement): 902-921. |
| [48] | Li F, Novotny R, Fialkowski M, et al. (2015) Determinants of young children's sleep duration in the US Affiliated Pacific: The Children's Healthy Living Program. The FASEB Journal 29(1 Supplement): 902-22. |
| [49] | Schulz AJ, Parker EA, Israel BA, et al. (1998) Conducting a participatory community-based survey for a community health intervention on Detroit's east side. J Public Health Manag Pract 4(2): 10-24. |
| [50] |
Israel BA, Schulz AJ, Parker EA, et al. (1998) Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health 19: 173-202. doi: 10.1146/annurev.publhealth.19.1.173
|
| [51] | Palafox NA, Buenconsejo-Lum L, Rikion S, et al. (2002) Improving health outcomes in diverse populations: competency in cross-cultural research with indigenous Pacific islander populations. Ethn Health 7(4): 279-285. |
| 1. | Shashank Kumar Singh, Amrita Chaturvedi, Leveraging deep feature learning for wearable sensors based handwritten character recognition, 2023, 80, 17468094, 104198, 10.1016/j.bspc.2022.104198 | |
| 2. | Ayush Tripathi, A. P. Prathosh, Suriya Prakash Muthukrishnan, Lalan Kumar, SurfMyoAiR: A Surface Electromyography-Based Framework for Airwriting Recognition, 2023, 72, 0018-9456, 1, 10.1109/TIM.2023.3248084 | |
| 3. | William Villegas-Ch, Isabel Urbina-Camacho, Joselin García-Ortiz, Detection of Abnormal Patterns in Children’s Handwriting by Using an Artificial-Intelligence-Based Method, 2023, 10, 2227-9709, 52, 10.3390/informatics10020052 | |
| 4. | Sike Ni, Mohammed A.A. Al-qaness, Ammar Hawbani, Dalal Al-Alimi, Mohamed Abd Elaziz, Ahmed A. Ewees, A survey on hand gesture recognition based on surface electromyography: Fundamentals, methods, applications, challenges and future trends, 2024, 166, 15684946, 112235, 10.1016/j.asoc.2024.112235 | |
| 5. | Jayakanth Kunhoth, Somaya Al-Maadeed, Suchithra Kunhoth, Younes Akbari, Moutaz Saleh, Automated systems for diagnosis of dysgraphia in children: a survey and novel framework, 2024, 27, 1433-2833, 707, 10.1007/s10032-024-00464-z | |
| 6. | Andrea Tigrini, Simone Ranaldi, Alessandro Mengarelli, Federica Verdini, Mara Scattolini, Rami Mobarak, Sandro Fioretti, Silvia Conforto, Laura Burattini, 2024, Automatic Handwriting Recognition with a Minimal EMG Electrodes Setup: A Preliminary Investigation, 979-8-3503-0799-3, 1, 10.1109/MeMeA60663.2024.10596883 | |
| 7. | Ayush Tripathi, Prathosh A.P., Suriya Prakash Muthukrishnan, Lalan Kumar, TripCEAiR: A multi-loss minimization approach for surface EMG based airwriting recognition, 2023, 85, 17468094, 104991, 10.1016/j.bspc.2023.104991 | |
| 8. | William Villegas-Ch, Jaime Govea, Angel Jaramillo-Alcazar, IoT Anomaly Detection to Strengthen Cybersecurity in the Critical Infrastructure of Smart Cities, 2023, 13, 2076-3417, 10977, 10.3390/app131910977 | |
| 9. | Andrea Tigrini, Federica Verdini, Mara Scattolini, Federico Barbarossa, Laura Burattini, Micaela Morettini, Sandro Fioretti, Alessandro Mengarelli, Handwritten Digits Recognition From sEMG: Electrodes Location and Feature Selection, 2023, 11, 2169-3536, 58006, 10.1109/ACCESS.2023.3279735 | |
| 10. | Ines Boutabia, Abdelmadjid Benmachiche, Ali Abdelatif Betouil, Chaouki Chemam, 2024, A Survey in the Use of Deep Learning Techniques in The Open Classroom Approach, 979-8-3503-5026-5, 1, 10.1109/PAIS62114.2024.10541268 | |
| 11. | Awang Karisma As’Ad Adi Asta, Eko Mulyanto Yuniarno, Supeno Mardi Susiki Nugroho, Cries Avian, 2023, Handwriting Classification based on Hand Movement using ConvLSTM, 979-8-3503-1395-6, 341, 10.1109/ISITIA59021.2023.10221037 | |
| 12. | Shashank Kumar Singh, Amrita Chaturvedi, An efficient multi-modal sensors feature fusion approach for handwritten characters recognition using Shapley values and deep autoencoder, 2024, 138, 09521976, 109225, 10.1016/j.engappai.2024.109225 | |
| 13. | Shashank Kumar Singh, Amrita Chaturvedi, 2025, Chapter 21, 978-3-031-81403-7, 288, 10.1007/978-3-031-81404-4_21 | |
| 14. | Mohammed A. A. Al-qaness, Sike Ni, TCNN-KAN: Optimized CNN by Kolmogorov-Arnold Network and Pruning Techniques for sEMG Gesture Recognition, 2025, 29, 2168-2194, 188, 10.1109/JBHI.2024.3467065 |
Marie K. Fialkowski, Ashley Yamanaka, Lynne R. Wilkens, Kathryn L. Braun, Jean Butel, Reynolette Ettienne, Katalina McGlone, Shelley Remengesau, Julianne M. Power, Emihner Johnson, Daisy Gilmatam, Travis Fleming, Mark Acosta, Tayna Belyeu-Camacho, Moria Shomour, Cecilia Sigrah, Claudio Nigg, Rachel Novotny. Recruitment Strategies and Lessons Learned from the Children’s Healthy Living Program Prevalence Survey[J]. AIMS Public Health, 2016, 3(1): 140-157. doi: 10.3934/publichealth.2016.1.140
| Architecture | Accuracy (%) |
| CNN-LSTM | 96.37 |
| CNN-2LSTM | 96.48 |
| CNN-GRU | 97.03 |
| CNN-2GRU | 96.29 |
DownLoad:
CSV
| Subjects | Accuracy (%) |
| Subject1 | 97.04 |
| Subject2 | 95.05 |
| Subject3 | 94.89 |
| All | 94.85 |
DownLoad:
CSV
| Architecture | Accuracy (%) |
| CNN-LSTM | 96.37 |
| CNN-2LSTM | 96.48 |
| CNN-GRU | 97.03 |
| CNN-2GRU | 96.29 |
| Subjects | Accuracy (%) |
| Subject1 | 97.04 |
| Subject2 | 95.05 |
| Subject3 | 94.89 |
| All | 94.85 |
| Method | Samples | Subjects | Classes | Accuracy (%) |
| Template Matching [13] | 350 | 6 | 10 | 97 |
| DTW [12] | 520 | 3 | 26 | 84.29 |
| DTW [14] | 780 | 4 | 26 | 78.24 to 92.42 |
| Proposed | 108k | 3 | 30 | 94.85 |
