Robust sign language detection for hearing disabled persons by Improved Coyote Optimization Algorithm with deep learning

1.   Introduction

Generally, disabled or special persons like deaf people are not able to talk with normal people. So, SL helps people who are unable to communicate ^[1]. In SL, many kinds of motions with numerous shapes are used. SL is one of the chief methods of communication among deaf as well as hearing people. SL detection networks are an effective method to talk with deaf and mute people ^[2]. A huge number of deaf and mute people exist across the world, and sometimes it becomes complex for common people to talk with them since not everybody can recognize SL. To create effectual communication among normal and disabled persons, there is a necessity to boost the usage of SL detection systems ^[3]. Non-verbal communication aids deaf as well as dumb people to talk between themselves or with others without any hassle. Deaf refers to a disability that harms a person's hearing capability and makes them unable to listen, whereas dumb refers to an incapacity that damages talking skills and makes them unable to talk ^[4]. If a person is not able to communicate or hear, it will be very hard to communicate with other persons. Due to this, SL plays a vital part. It allows an individual to talk without words ^[5]. However, one of the main issues that exist in this is that not many people know SL. Dumb and deaf people can be capable of talking between themselves by employing SL, but with normal hearing people it will be difficult for them to communicate due to lack of SL knowledge. This problem can be fixed by the usage of technique-driven solutions ^[6]. By employing a solution, people can effortlessly convert SL gestures into general spoken language.

SL detection denotes to usage of procedures and models employed to detect the resulting series of gestures and clear their meaning in a manuscript or language ^[7]. This technique contains numerous research areas, like pattern recognition, human-computer interaction (HCI), natural language processing (NLP), CV, video acquisition and processing, etc. It is a challenging topic with high difficulty ^[8]. Currently, typical SL detection systems are primarily separated into two kinds, namely machine vision-based schemes and sensor-based schemes. Automated recognition of human signs is a difficult multi-disciplinary problem that is not completely determined ^[9]. Presently, numerous models have been proposed including ML techniques for SL recognition. Many efforts have been made to detect human signs over the development of DL methods ^[10]. A system that is based upon DL models handles architecture, while the learning process is stimulated physically with traditional networks.

This manuscript develops an SL detection via an Improved Coyote Optimization Algorithm with DL (SLR-ICOADL) technique hearing disabled persons. The goal of the SLR-ICOADL technique is to accomplish an accurate detection model that enables communication for persons using SL as a primary case of expression. At the initial stage, the SLR-ICOADL technique applies a bilateral filtering (BF) approach for noise elimination. Following this, the SLR-ICOADL technique uses Inception-ResNetv2 for feature extraction. Meanwhile, the ICOA is utilized to select the optimal hyperparameter values of the DL model. Finally, the extreme learning machine (ELM) classification model can be utilized for the recognition of various kinds of signs. The experimental outcomes highlight that the SLR-ICOADL technique gains promising performance in the SL detection process.

2.   Literature works

Al-onazi et al. ^[11] presented an Arabic SL Gesture Classification employing the Deer Hunting Optimizer with ML (ASLGC-DHOML) technique. This method mainly pre-processed the input gesture images and produced feature vectors through the DenseNet169 framework. For gesture detection and classification, an MLP algorithm was utilized for recognizing and categorizing the present SL gestures. Also, the DHO method was implemented for parameter optimization of the MLP method. In ^[12], an Indian SL recognition technique was developed by applying ML methods. This developed approach creates the usage of different images of individuals, which represent the alphabet in Indian SL. Such images are further pre-processed, and then the technique employs these achieved images in testing and training the ML method.

Aliyev et al. ^[13] implemented object recognition and classification by leveraging pre-trained lightweight CNN methods. Initially, a database including close to a thousand images could be gathered, and then stimulating objects on images were labeled with bounding boxes. For designing, training, analyzing, and employing the applicable model, the TensorFlow Object identification API with Python was implemented. A pre-trained MobileNet-v2 algorithm was leveraged for tasks. In ^[14], an Enhanced Bald Eagle Search Optimization with TL SL Recognition (EBESO-TLSLR) method was developed. In this model, the SqueezeNet architecture was utilized for feature map generation. To recognize the SL types, the long short-term memory (LSTM) model was employed. Further, the EBESO technique was utilized for optimal hyperparameter selection of the LSTM model.

In ^[15], a solution was introduced with the utilization of a ML technique, which can recognize hand gestures and convert them into text or speech and conversely transform from speech or text to gestures. A webcam must be employed for capturing the region of interest (RoI), like identifying the hand movement and gestures to be represented. According to the identified gestures, the noted recording can be played. In ^[16], a method was developed that can employ a visual hand database dependent upon Arabic SL as well as analyze this visual information in textual data. The database employed includes Arabic SL alphabets, and all categories signify various meanings by their hand signs or gestures. Numerous data augmentation and preprocessing methods were implemented for the images. The analysis was carried out by employing diverse pre-trained methods using the specified database.

The authors of ^[17] considered a new supervised learning technique for ISL identification, which was beyond standard classification approaches. This algorithm utilizes advanced models devised for classifying novel interpretations. With the help of a wide-ranging database, complex patterns and nuances have been recognized, promoting correct and adaptable decision-making. A CNN method was implemented, not only in data classification but also for refinement of classification limitations and iterative learning. Marais et al. ^[18] examined signer-independent SLR with the LSA64 database and considered various feature extraction methods. These 2 techniques have been designed as an InceptionV3-GRU framework that employs raw images as input as well as a pose evaluation LSTM model. MediaPipe Holistic was applied to remove pose evaluation landmark coordinates. Lastly, the third architecture implements augmentation and TL through the pose analysis LSTM method.

3.   The proposed method

In this manuscript, a novel SLR-ICOADL methodology is developed for hearing disabled persons. The goal of the SLR-ICOADL technique is to accomplish an accurate detection model that enables communication for persons using SL as a primary case of expression. Figure 1 demonstrates the entire procedure of the proposed SLR-ICOADL methodology.

3.1.   Preprocessing

Initially, the SLR-ICOADL technique applies the BF approach for noise elimination. The BF system is a non-linear image filter system utilized for noise removal while maintaining fine and edge details ^[19]. Different from linear filters that execute a weighted average in a neighborhood of pixel values, BF considers either the spatial or intensity differences among pixels. It allocates weights depending on either spatial distance or intensity similarity, ensuring that pixels with the same intensity give more to the filtering method. In the context of noise elimination, BF is particularly effective at smoothing images while maintaining sharp transitions at edges. This can be accomplished by decreasing the control of various pixels with distinct intensities, thus avoiding blurring across edges. The BF adaptability makes it useful for different kinds of noise comprising salt-and-pepper noise, as it selectively smooths regions with consistent intensity while maintaining edges and details.

3.2.   Feature extractor

At this stage, the SLR-ICOADL technique uses Inception-ResNetv2 for feature extraction, which integrates inception as well as residual connection techniques for enhanced performance ^[20]. This hybrid method allows the system to make use of the advantages of both approaches, with a quicker training period and averting vanishing gradient issues. In addition, residual links permit the system to skip a few layers at the time of training. Also, Inception-ResNetv2 uses multiple kernel sizes in one layer to extract outlines with assorted orders, enhancing the network's capability to catch features of variable difficulty.

The Inception-ResNet technique has numerous blocks that contain filter concatenation, convolutional layers, ResNet, ReLU activation functions, and inception structures. Its unique design and effective usage of filters has produced notable results in medical image detection challenges. Therefore, we use the InceptionResNetV2 pre-trained technique as the base for our research.

The Inception-ResNetv2 technique is normally used in CT-scan image detection due to its capability to remove important features from medical images. This DL framework integrates Inception and ResNet, two well-known DL techniques. Inception methods are useful due to their ability to remove features at manifold measures, whereas ResNet approaches efficiently solve the problem of vanishing gradients at the time of training. The Inception-Resnetv2 technique is personalized to overcome the limits of these methods to achieve high accuracy in image detection challenges. Furthermore, this technique effectively handles huge datasets with decreased calculation time, creating a desired method for the analysis of medical images.

3.3.   ICOA-based hyperparameter tuning

In this work, the ICOA is utilized to select the optimal hyperparameter values of the DL model. The COA is a novel optimization model stimulated by the performance of coyotes in the environment and the traditional exchange between coyotes ^[21]. The coyote is a clever mammalian animal that resides mostly in North and Central America. For example, in the large applications, scientists tried to clear coyotes in the west, but rather this animal extended into the east and their populace increased rather than declining.

In the COA, the social performance of coyotes is measured as a cost function, given by

where SOC specifies the charging rate, $p$ defines the group, and $t$ designates the time of simulation.

Initially, a few arbitrary coyote candidates are designated as solution candidates in this space. This procedure is expressed as

where $r\in \left[\mathrm{0, 1}\right]$ are random values, $U{r}_{j}$ defines the upper range, and $L{r}_{j}$ denotes a lower range of the $j$th variable in the hunt space.

An objective function for every coyote is given as

Here, groups' places can be upgraded arbitrarily by traveling from one cluster to another. This procedure's formula is given by

where $Nc$ represents the number of coyotes as a candidate solution that is lesser or equivalent to $\sqrt{200},$ and ${P}_{1}$ has a value higher than 1. The cluster populace for coyotes was restricted to 14 to improve model diversity.

An optimal candidate is defined as an alpha coyote, which is demonstrated by the formula

A culture transformation for coyotes is given by

Here, ${R}^{p, t}$ represents coyotes, social condition position for cluster number $p$ at time $t$ for variable $j.$

Further, a coyote's life cycle depends on a mixture of social performance and environmental factors of parent coyotes, arithmetically exhibited as

where ${r}_{j}$ refers to random values from the range zero and one$,$ ${\sigma }_{j}$ defines random solutions from the design variable limit, ${r}_{2}$ designates a random coyote in cluster $p, {j}_{1}$ and ${j}_{2}$ signify random design variables, $p{r}_{a}$ describes the association, and $p{r}_{s}$ refers to scatter probabilities that announce coyote's national variety in group attained by

where $d$ signifies the dimension of a variable.

The balancing procedure $\left(Ble\right)$ for the life cycle is clarified in the following result: If the quantity of coyotes in the group is one, $Ble$ survives and the worst outcomes of coyotes expire. If the quantity of coyotes in the cluster is higher than 1, $Ble$ exists and the oldest worst outcome of coyotes expire. Otherwise, $Ble$ deceases. The possibility of dying for $Ble$ is 15%. Figure 2 illustrates the steps involved in the COA.

Two aspects are employed for defining cultural transition among clusters. These two aspects are given as

where ${\delta }_{1}$ denotes cultural differences between the alpha and nominated coyote $\left(cr1\right)$, and ${\delta }_{2}$ defines culture dissimilarity among the cluster culture and nominated coyote $\left(cr2\right)$.

The upgrading procedure for social performance depends on the leader and group effect measured by

where ${r}_{1}$ and ${r}_{2}$ define random values between $0$ and 1.

Lastly, the novel cost of upgrading is given by

The COA shows a better solution for the optimizer problem; however, there is a major drawback to this method. The ICOA presents two classical approaches to resolving early convergence problems.

The first mechanism is to enhance the COA and deal with the inclination speed to accomplish a better solution. The primary goal of COA is to model the updating location to control the random value. In the COA, the early individual moves from the high step size to create a balance among the local and global search, and the step size is decreased by the local search from the solution space at the last iteration:

where $\left(so{c}_{c}^{best}\right)$ and $f\left(so{c}_{c}^{worst}\right)$ are the worst and best performances of the main function, respectively.

The self‐adaptive weighting model enhances the value of weight for the social behaviors of the coyotes by taking the alpha value to decrease the variance amongst the worst and best performances. The chaos conception is the 2nd process used to resolve the local optimization problems. Also, this helps to decrease the time complexity.

The ICOA model creates pseudo‐random values that have sequence and ergodic random features. The logistic map has been used as a typical chaotic mechanism and is used for the three parameters $(\eta, {r}_{1}, {r}_{2})$. The formula can is given by

where the initial value ${\varnothing }_{1}$ is a randomly generated number within [0, 1] and, ${\varnothing }_{k}$ denotes the value of ${i}^{th}$ chaotic iteration.

The fitness choice is a significant aspect of manipulating the solution of the ICOA approach. The hyperparameter choice model contains the outcome of the encoder system for evaluating the solution of candidate results. During this case, the ICOA methodology assumes accuracy as the main condition to plan the fitness function (FF) that is expressed as

where $FP$ and $TP$ denote the false and true positive values, respectively.

3.4.   Classification using ELM model

Next, the ELM classification model can be utilized for the recognition of various kinds of signs. The ELM technique simplifies one-layer backpropagation (BP) systems ^[22]. Dissimilar to conventional learning models, ELM arbitrarily picks input weights as well as biases of hidden layer (HL) neurons which can make these limits beforehand looking at learning data. ELM neural network presented as a simplified single HL with feed-forward neural network (SLFN).

SLFN is expressed as

where ${v}_{j}$ are HL neurons $(i = 1, \dots, n),$ ${a}_{i}$ are the weight of connections among input variables and neurons in HL, ${w}_{jo}$ are the weight of connections amid neurons in HL and output neurons, ${b}_{i}$ are the bias of neurons in HL, ${b}_{0}$ are the bias of output neurons, ${f}_{i}$ represents the activation function of neurons, $g$ represents the activation function of output neurons, and ${s}_{i}$ defines dual variables.

The ELM output function is given as

where ${\sum }_{i = 1}^{L}{\beta }_{1}{h}_{1}\left(x\right) = h\left(x\right)\beta$ is the output weight vector, and ${\sum }_{i = 1}^{L}{\beta }_{1}{h}_{1}\left(x\right) = h\left(x\right)\beta$ represents a non-linear feature of a learning machine. The ${h}_{1}$ formula is given by

where $G({a}_{i}, {b}_{i}, x)$ is a continuous non-linear function.

4.   Results analysis

In this section, the performance analysis of the SLR-ICOADL technique is tested using the SL dataset ^[23]. The dataset includes samples under 29 classes. For experimental validation, the training/testing dataset is split 80:20. In Table 1 and Figure 3, the overall SL detection results of the SLR-ICOADL technique are reported in terms of distinct metrics. The results highlight that the SLR-ICOADL technique recognizes various kinds of signs proficiently. In addition, it is noticed that the SLR-ICOADL technique reaches average $pre{c}_{n}$, $rec{a}_{l}$, $acc{u}_{y}$, and ${F}_{score}$ of 99.97%, 99.97%, 99.96%, and 99.98%, respectively.

Figure 4 shows the accuracy and loss curves of the SLR-ICOADL technique compared with other existing approaches. The simulation values imply that the accuracy value inclines to enhance and loss performance inclines to minimal with an enhancement in epoch counts. It is also apparent that the training loss is minimal and the validation accuracy is superior on the test database.

Table 2 illustrates the comparative analysis of the SLR-ICOADL technique with other recent methods ^[24]. Figure 5 demonstrates the $pre{c}_{n}$ and $rec{a}_{l}$ analysis of the SLR-ICOADL methodology compared with recent algorithms. The simulation values show that the SLR-ICOADL technique achieves better performance. Based on $pre{c}_{n}$, the SLR-ICOADL method has a higher $pre{c}_{n}$ of 99.97%, while the ODTL-SLRC, SGD optimizer, RMSProp optimizer, and Adam optimizer models have achieved lesser $pre{c}_{n}$ of 99.93%, 99.90%, 99.92%, and 99.89%, respectively. In addition, based on $rec{a}_{l}$, the SLR-ICOADL methodology reached a maximum $rec{a}_{l}$ of 99.97%, while the ODTL-SLRC, SGD optimizer, RMSProp optimizer, and Adam optimizer techniques have a lower $rec{a}_{l}$ of 99.93%, 99.90%, 99.89%, and 99.84%, respectively.

Figure 6 represents the $acc{u}_{y}$ and ${F}_{score}$ outcomes of the SLR-ICOADL method compared with other algorithms. The experimental values imply that the SLR-ICOADL technique achieves better performance. Based on $acc{u}_{y}$, the SLR-ICOADL methodology has reached a superior $acc{u}_{y}$ of 99.96%, whereas the ODTL-SLRC, SGD optimizer, RMSProp optimizer, and Adam optimizer approaches have accomplished minimal $acc{u}_{y}$ of 99.93%, 99.90%, 99.75%, and 99.25%, respectively. Additionally, based on the ${F}_{score}$, the SLR-ICOADL system has a maximum ${F}_{score}$ of 99.98%, while the ODTL-SLRC, SGD optimizer, RMSProp optimizer, and Adam optimizer methodologies achieved a lower ${F}_{score}$ of 99.94%, 99.84%, 99.88%, and 99.89%, respectively.

Table 3 examines the overall recognition rate (RR) and computation time (CT) outcomes of the SLR-ICOADL technique with existing models. In Fig. 7, a comparison study of the SLR-ICOADL technique is performed in terms of RR. Based on RR, the SLR-ICOADL technique gains a maximum RR of 99.95%, while the KNN, SVM, ANN, CNN, and ODTL-SLRC techniques attain lower RR values of 96.25%, 98.10%, 98.11%, 99.89%, and 99.93%, respectively.

A brief computational time (CT) result of the SLR-ICOADL technique compared with recent approaches is shown in Figure 8. The results indicate that the KNN, SVM, ANN, and CNN models have resulted in ineffectual CT values of 16.54 min, 14.34 min, 15.41 min, and 11.21 min, respectively. Although the ODTL-SLRC technique offers a reasonable CT of 6.44 min, the SLR-ICOADL technique reported the lowest CT of 3.90 min. These results demonstrate the enhanced SL recognition results of the SLR-ICOADL technique.

5.   Conclusions

In this manuscript, a new SLR-ICOADL methodology was developed for hearing disabled persons. The goal of the SLR-ICOADL technique is to accomplish an accurate detection model that enables communication for persons using SL as a primary form of expression. Primarily, the SLR-ICOADL technique applies the BF approach for noise elimination. Following this, the SLR-ICOADL technique uses Inception-ResNetv2 for feature extraction. Then, the ICOA is utilized to select the optimal hyperparameter values of the DL model. Finally, an ELM-based classification model can be utilized for the recognition of various kinds of signs. To demonstrate the better performance of the SLR-ICOADL algorithm, a detailed set of experiments were performed. The experimental outcomes demonstrate that the SLR-ICOADL technique has promising performance in the SL detection process.

Author contributions

All authors of this manuscript contributed equally.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The authors extend their appreciation to the King Salman Center for Disability Research for funding this work through Research Group no KSRG-2023-420.

Conflict of interest

The authors declare that they have no conflict of interest. The manuscript was written through the contributions of all authors. All authors have approved the final version of the manuscript.