A sophisticated Drowsiness Detection System via Deep Transfer Learning for real time scenarios

1.   Introduction

Road safety is an issue that deserves more attention since it represents one of the great opportunities to save more lives around the world. In accordance with the World Health Organization (WHO) factsheets, road accidents are the eighth reason of mortality globally, while it is fourth in the United States. Each year, roughly 1.3 million people die due to road traffic crashes, that is, one person dies every 25 seconds on average ^[1]. In addition, road traffic injuries are the first leading cause of death for children and adolescents aged between 5 and 29 years. Besides, about 50 million people suffer from in-juries and as a result many endure disabilities. Nevertheless, all those deaths and injuries are preventable. The critical reason in the crash chain is assigned to the driver. Approximately 94% of all car accidents are caused by human errors ^[2]. According to the American National Highway Traffic Safety Administration (NHTSA), the major cause of fatal crashes is the driver state related factor: Drowsiness, alcohol and speed ^[2].

Drowsy driving is the act of driving while tired and feeling fatigued or sleepy. Indeed, driver drowsiness is a hazardous state that occurs unintentionally and can lead to fatalities, but it is difficult to detect. Driver drowsiness is one of the major factors responsible for crash severity, especially for vulnerable road users ^[3]. The American National Safety Council (NSC) declares that each year, drowsy driving counts about One Hundred Thousand crashes, Seventy Thousand injured persons, and more than One Thousand Five Hundred mortalities ^[4]. It contributes to an estimated 9.5% of all crashes based on the statistics of the American Automobile Association (AAA) Foundation for Traffic Safety ^[5].

It is possible to notice the drowsiness state of a driver in its initial phase and keep the driver alert in order to get over a potential crash using DDD systems that monitor and detect the driver's behavior and especially drowsiness ^[6]. In addition to the obvious need, the integration of these systems will become compulsory for automobile manufacturers in the European Union (EU) soon ^[7]. Indeed, the European Commission (EC) decided to include DDD as one of the safety mandates in cars sold in the European Union ^[7]. In addition to that, the European New Car Assessment Program (Euro NCAP) will incorporate drowsiness detection as part of its assessment protocol when evaluating a vehicle's safety ratings in the 2025 Roadmap ^[6].

Several researchers have recently considered the automatic DDD systems. Three major categories of measures that have been used widely for these systems are ^[8,9,10]:

(1) Vehicle measures: Some parameters like deviations from lane position, pressure on the acceleration pedal and the steering wheel shaking are regularly supervised. Any changes in these parameters that surpass a particular threshold is very likely linked to a drowsy driver ^[11,12].

(2) Behavioral measures: The conduct of the driver especially facial expressions (yawning, eyes' closure, eyes' blinking), head pose, body and hand movements are monitored through a camera or visual sensors and the driver is notified if any of these drowsiness signs are detected ^[13,14,15].

(3) Physiological measures based on physiological signals. The evaluation of the cognitive and concentration signs in response to the perceptual stimulus is doable using physiological measurement signals such as Electro-oculogram (EOG), Electromyogram (EMG), electrodermal activity (EDA), Photoplethysmogram (PPG), Electrocardiogram (ECG) and Electroencephalogram (EEG) signal which is the most used ^[9,10,16].

In this work, we will develop a sophisticated hybrid DDD system to recognize accurately and efficiently the drowsiness behavior of a driver in order and to alert him depending on both the eyes closure and the mouth opening ratios and using a combination of images issued from car camera and deep learning (DL) techniques. The proposed approach comprises two processes. An offline process is dedicated to the learning module of three CNN networks based on a dataset. We enhanced this dataset using appropriate data augmentation techniques to achieve the classification task with good accuracies. An online process is devoted to the detection module and the prediction module. Indeed, a car cam that records real-time images of the driver is used. The driver's facial landmarks will be limited in a frame using image recognition techniques. We will determine the Eye Closure Ratio (ECR) and the Mouth Opening Ratio (MOR) using the Chebyshev distance for the first time in literature to the best of our knowledge. This distance gives the best result in terms of accuracy compared to other distance measures. The driver drowsiness state is detected based on the ECR or/and the MOR values. Sometimes, and due to environmental factors, some false drowsy states causing false alerts can be detected. This can be intrusive and annoy drivers that may end up not believing them anymore. Thus, in order to evade falsely detected drowsy states, we use the pretrained CNN models to identify the real situation of the driver. The fusion of these predictors, based on ensemble learning methods, will determine whether the driver is tired. Concisely, the novelty of this DDD system lies in; (ⅰ) the use of the Chebyshev distance, (ⅱ) the ability to detect the falsely detected drowsy states and (ⅲ) the use of ensemble learning method to accurately recognize the drowsiness state. The proposed DDD system is characterized by its cost-efficiency, ease of use, non-invasiveness, and automatic functionality. These features collectively contribute to its seamless integration into industrial processes, ensuring minimal disruption and maximum efficiency.

Thus, this paper is organized as follows. In Section 2, we will detail concepts in connection with the proposed DDD system including the drowsiness definition, the facial behavioral measures and transfer for DDD systems and some research studies related to that. Section 3 introduces the proposed approach, methodology and materials, it involves the offline process representing the learning module and the online process displaying the detection module. In Section 4, we will detail experimental results showing the performances metrics and some simulations figures related to the proposed system. Section 5 focuses on the proposed method and provides in-depth discussions and comparisons with other existing works. This section highlights the promising nature of our method in terms of both efficiency and accuracy, particularly when compared to other approaches. These findings contribute to establishing the credibility and potential of our proposed method in the field. Finally, in Section 6, the paper ends with a comprehensive and thought-provoking conclusion that summarizes the key findings, implications, and potential avenues for future research.

2.   Background & related works

2.1.   Drowsiness

Drowsiness is defined as the transition between wakefulness and sleepiness. If a person falls asleep when driving, he will lose control of the car steadily, which often leads to a car crash. Therefore, drowsy driving, whether it is due to fatigue, medication or a sleep disorder, is a serious issue. In order to obviate these situations, the state of a drowsy driver should be detected accurately and early. DDD systems based on face recognition have a great importance in detecting and alerting the driver to avoid fatal situations, especially, when combined with DL techniques. In our work, we focus on the study of DDD systems based on facial expressions measures.

2.2.   Facial expressions' behavioral measures for DDD systems

Studying the features of the driver's physical behavior represents a good method to detect more efficiently the driver's drowsiness. DDD systems are generally classified into three types, based on the mouth, head and eye movements.

Table 1 enumerates some of the facial expressions' behavioral measures ^[17].

There are many DDD systems which are based on facial expressions. They use many and diverse parameters and methods to conceive their detection procedure.

Eye state: the eye state is one of the relevant methods among behavioral measures to detect driver drowsiness. Features like the eye-opening rate, the eyelid distance and PERCLOS are among the top indicators of the driver's drowsiness.

Khan et al. ^[26] proposed a real-time DDD system that utilized eyelid closure as a key indicator. The technology employed surveillance videos to monitor whether the driver's eyes were open or closed. The system began by identifying the driver's face, followed by locating the eyes and assessing the curvature of the eyelids using an extended Sobel operator. By determining the concavity of the curve, the system classified the eyelids as either open or closed. If the eyes remained closed for a fixed duration, an alarm was triggered. The system was evaluated using three datasets. The first dataset achieved an accuracy of 95%, the second dataset attained 70% accuracy, and the third dataset surpassed 95% accuracy.

A sleepiness detection technique was created by Maior et al. ^[27] based on the eyes' movements using a basic web camera. The EAR metric was used to calculate the blinking period of time. To deter-mine the EAR value, the ratio between an eye's height and width is computed. When the EAR value is high, the eye is open, and when it is low, the eye is closed. The idea consists of three steps, the EAR computation, the blink classification, and the real-time detection of potential drowsiness state. The EAR were measured and saved at each frame. Unless the blink period is smaller than a specific one, drowsiness will be indicated. Accuracy attained was 95%.

Zandi et al. ^[28] proposed the use of eye tracking data as a non-intrusive method for detecting driver drowsiness. In their study, this data was collected from 53 drivers during a simulated driving experiment. First, multichannel EEG signals were recorded to serve as the baseline reference. For the analysis, a binary classification approach was employed, utilizing both a Random Forest (RF) and a Support Vector Machine (SVM) classifier. Different lengths of eye tracking epochs were chosen for feature extraction, and each classifier performance was evaluated for every epoch length. The results showed that the RF classifier achieved an accuracy of 88.37% to 91.18% across all epoch lengths, outperforming the SVM classifier, which achieved an accuracy of 77.12% to 82.62%.

A real-time DDD system based on the eye closure using DL was proposed by Hashemi et al. ^[29]. For the classification of eye closure, a Fully Designed Neural Network (FD-NN) along with Transfer Learning using the VGG16 and VGG19 models, were developed. To enhance the training data, the authors supplemented the dataset with 4157 new photographs. FD-NN achieved an accuracy of 98.15%. Furthermore, the Transfer Learning approach with the VGG16 model achieved an accuracy of 95.45%, while the VGG19 model achieved an accuracy of 95%.

Mouth state: Mouth state is used to predict driver drowsiness in real-time driving situations in numerous studies. Alioua et al. extracted features from mouth movements using an SVM and a method based on the Circular Hough Transform (CHT) for DDD system ^[30]. These systems were effective in real-time scenarios in various lighting environments. Based on real images, the experiment's findings showed that yawning could be detected with an accuracy rate of 81%.

Xiaoxi et al. ^[31] performed a DDD system based on CNN using depth video sequences to detect driver fatigue during the nighttime. Indeed, two CNNs were used: A spatial CNN that locates objects, and a temporal CNN that searches for information between adjacent frames. These two CNNs enable the system to calculate motion vectors at each frame, allowing it to detect yawns even when the driver covers his or her mouth with a hand. Experiments display an accuracy of 91.57%.

Hybrid behavioral features (eye and mouth states): In recent years, studying DDD systems has been generally based on hybrid behavior analysis. Savasx et al. used a CNN model to detect drowsiness based on both mouth and eye states ^[32]. The authors showed how to use PERCLOS data and mouth yawning frequency to detect driving drowsiness in real-time. These data were trained with various CNN for the identification of drowsiness factors. The results display an accuracy of 98.81%.

Celecia et al. ^[33] suggested an economical and accurate DDD system. Images were recorded on camera with an infrared illuminator. The system's process incorporates the obtained features from the eyes and mouths using a Raspberry Pi 3 Model B. The features states were done through a cascade of regression tree algorithms and then they were combined by a Mamdani fuzzy inference system, in order to predict the state of the driver. The system produces a final output that indicates three degrees of drowsiness. The study resulted in a DDD system with 95.5% accuracy resilient to various ambient illumination situations.

A non-intrusive method that can identify a real state of drowsiness was presented by Alioua et al. ^[34]. Closed eyelid and open mouth recognition techniques are used to identify drowsiness based on images collected from a webcam. To determine the face region in each image, the system used an SVM face detector. The Hough Transform was employed to locate the mouth and eyes' regions in the face. Based on the extracted features, the system made determinations regarding the driver's drowsiness. The results confirmed the system's robustness, achieving 94% accuracy and 86% kappa statistic value.

In this work, we perform the proposed DDD system according to facial expressions based on both eyes and mouth states. The ECR and the MOR are calculated utilizing the Chebyshev distance in contrast to the latest works which exclusively used the Euclidean distance. The Chebyshev distance provides better accuracy especially in cases where the operational speed is inherent. The proposed DDD system can recognize a real-time drowsy situation with high performances thanks to the synergy between the offline and the online module of the proposed approach. Indeed, the fusion of DNNs decides the real situation of the driver by overthrowing the falsely detected drowsy states.

2.3.   Transfer learning for DDD systems

Euclidean distance. Deep Learning (DL), an area of Machine Learning (ML) community, is one of the most important re-search trends thanks to its considerable success in many domains. One of the benefits of DL networks is the ability to learn large amounts of data that allow us to achieve excellent results for many complex cognitive tasks. Among the most popular DL networks, there is a definite type called convolutional neural networks (CNNs). The main advantage of CNNs is their capability to find patterns automatically and to detect the important features among images without any human guidance that made it the most widely used ^[35,36]. A CNN architecture is represented in Figure 1.

A pretrained CNN model is a model that has been trained on a large dataset to learn a specific task, such as image recognition or natural language processing, and then saved for later use. There is a wide range of pre-trained models that are available, such as Inception, Xception, VGG and ResNet families. The choice of the suitable model for one task is based on the domain of the application, the task at hand and the availability of data. Otherwise, deep transfer learning requires a significant correlation between the data of the pre-trained model and the target task so that they are matching ^[37].

A relevant concept linked to CNNs is transfer learning ^[37]. This promising technique consists of using a powerful pre-trained CNN model, called Deep Neural Networks (DNNs), to solve a different task of a similar domain, and use it to resolve a new task ^[38]. Transfer learning means that training does not require high amounts of data and does not need to be started over for every new task. Thus, transfer learning saves both resources and time ^[39]. The lesser training time leads to better predictive performance and initializing the network with pretrained structures improves the generalization even after considerable fine-tuning to the used dataset ^[40].

Many studies using CNNs based on the DL techniques have been presented to detect drowsy drivers. The study in ^[41] analyzed the driver images using a CNN with VGG16 classification algorithm to determine if the eyes of the driver are open or closed and if he is yawning. The VGG16 model attained 91 % of accuracy and an F1-score of over 90 % for each class.

In order to detect tiredness, Dua et al. ^[42] suggested an architecture of four DL models that use RGB videos of drivers as input. Based on an ensemble process, the output of these models is combined to produce the final output, which is determined by a SoftMax classifier. 85% of accuracy was attained. Yu et al. ^[43] proposed a framework for the driver drowsiness detection based on 3D-deep convolutional neural network. The recognition of driver's drowsiness status was done using the condition-adaptive representation with an accuracy of 76, 2 %. Park et al. ^[44] proposed a DL network using three deep neural networks to access the driver facial features. For the DDD system, the three networks' outputs are pooled and incorporated into a Softmax classifier. Results from the experiment have a detection accuracy of 73.66%. Abbas ^[45] designed a HybridFatigue system, which combines both visual and non-visual features to detect driver fatigue. The visual features were measured through PERCLOS to calculate the degree of eye closure. The non-visual features were obtained using ECG sensors, providing physiological data to complement the visual information. To process and analyze these hybrid features effectively, a multi-layer-based transfer learning approach was employed. This approach utilized a combination of CNNs and deep-belief network (DBN) models. This system achieved a detection accuracy of 94.5% when tested on 4250 images.

3.   Proposed approach

3.1.   Description and objectives of our research

In this section, we introduce the considered DDD system to detect driver drowsiness from different driving scenarios based on pretrained CNNs using transfer learning techniques. The DDD system con-sists of two processes and three modules: an offline process that lies in the learning module and an online process that involves the detection module and the prediction module.

The key contributions of the proposed approach are as follows:

● We propose a novel DL approach that can automatically detect and estimate the driver's drowsiness based on camera and deep transfer learning methods.

● We utilize the Chebyshev distance to check the eyes and the mouth states (open or closed) based on facial landmarks to detect drowsiness.

● We generate data augmentation techniques for the used dataset in order to enlarge and enrich the data towards improving the training procedure.

● We classify the detected drowsiness state by three pretrained CNN models, which effectively improve the performance of the DDD system.

● In order to obtain better recognition performance, we refer to the fusion of classifiers by combining all the models' outputs to find the final prediction based on the ensemble learning theory (the bagging process).

● The proposed system is able to avoid falsely detected drowsiness situations and to alert only the real ones.

3.2.   Offline process (Learning module)

The offline process consists of training three CNN models; the VGG16, the VGG19 and the Res-Net50. These models represent the most object identification accuracies ^[46]. They will be used later to decide if the driver is drowsy or not for a real-time detected drowsiness state.

3.2.1.   Dataset used

In this study, the YAWDD dataset ^[47] was used in the offline process for the classification task of driver drowsiness. This data set is composed of 2900 samples with various facial features of both male and female drivers recorded by an in-car camera. It is divided into four categories: Yawn, no-yawn, open eye and closed eye. Figure 2 shows the image samples of the open eye, closed-eye, yawn and no-yawn classes from the dataset. The samples are almost equally distributed on the four classes.

3.2.2.   Training

In this study, the YAWDD considers DL advantages, and we applied deep neural networks with multiple hidden layers for driver drowsiness detection on the presented dataset. Three DNNs were considered for training:

● VGG16 is a pretrained CNN model proposed in 2014. The model attains 92.7% test accuracy in ImageNet that is a dataset of more than 14 million images including 1000 classes ^[48].

● VGG19 is a pretrained CNN model that is trained using more than one million images from the ImageNet database. The network covers 19 layers and can classify images into 1000 classes ^[48]. VGG16 and VGG19 are both developed by the Visual Graphics Group (VGG) at the University of Oxford. They are known for their simplicity and uniform architecture. VGG models use small 3x3 convolutional filters with a stride of 1 and max-pooling layers to down sample the spatial dimensions.

● ResNet50 is a widely used ResNet model developed by Microsoft Research. It is a pretrained CNN with 50 deep layers on top of each other to build a network ^[49]. Like VGG16 and VGG19, ResNet50 is also pretrained on the ImageNet dataset for image classification tasks. ResNet50 introduces skip connections, or shortcuts, to the traditional deep network architecture, which allows the model to learn residual functions. This helps in training deeper networks without the vanishing gradient problem.

The main difference between these two families of model lies in the architecture and approach to learning. While VGG models focus on stacking more layers, ResNet50 emphasizes learning residual functions to address the challenges of training very deep networks.

3.2.3.   Data augmentation

As they have many tunable parameters, the performance of DNNs depends on the quality and quantity of training data. In fact, these DNNs need a large amount of data in the training phase in order to perform a complex task with high accuracy. However, the dataset used for the training is reduced. Therefore, we refer to data augmentation techniques aiming to artificially increase the quantity of data by producing new data points from available data. This includes making small transformations to the original data by making simple alterations on image data such as geometric transformations, color, and color space transformations. Figure 3 represents an overview of the training process.

3.3.   Online process (Learning module)

3.3.1.   The detection module

A simple car camera is placed on the top of the vehicle. The face region of the driver is detected from live video. The identification of the eyes and mouth landmarks are obtained by the application of Dlib toolkit. The ECR and the MOR, calculated using the coordinates of landmarks, are presented to identify respectively the state of eyes and the mouth If the eyes are closed or/and she/he is yawning, a drowsiness state is detected. The detection module mechanism is given in Figure 4.

Identification of facial landmarks: The driver video is introduced directly to the Dlib ^[50] library frame by frame. Dlib toolkit is an open-source library using C++ language and including machine learning algorithms and tools used in different applications and in many domains. We used Dlib to identify the driver's face's essential features after extracting the driver's face from the image at each frame. A facial landmark detector with 68 face-specific coordinate points is offered by the Dlib package. Viola and Jones ^[51] proposed this machine learning-based detection technique.

Kazemi et al. ^[52] proposed a technique that estimates the positions of facial landmarks on images. This technique can be used for real-time facial features detection including the eyes, eyebrows, nose, ears, and mouth. This detector identifies 68 major facial features positions, as shown in Figure 5.

Based on the index of the face sections, we can extract the specific facial structures. Thus, we can detect and access both the eye and mouth regions according to the following facial landmark index:

● The right eye: (36, 42),

● The left eye:(42, 48),

● Mouth: (49, 68).

In this work, we used 32 feature marks characterizing the left eye, right eye, and mouth to calculate eye closure and mouth opening. We calculated the ECR and the MOR, using the Euclidean distance and the Chebyshev distance. The Chebyshev distance performs better than the Euclidean distance. Hence, we refer to the use of the Chebyshev distance.

The Chebyshev distance, contrary to the Euclidean distance, is typically used in very specific use cases which makes it difficult to use as an all-purpose distance. It is generally used when the implementation speed is critical. The main benefit of using the Chebyshev distance over the Euclidean distance is that it takes less time to choose the distances between the pixels. It is the most appropriate distance used in cases where the execution speed is critical ^[53].

The Chebyshev distance (Figure 6) is a metric in which the distance between two vectors is the greatest of their differences. The Chebyshev distance guarantees that the next considered points are potentially placed at the border of the region of point in one dimension, and these points usually discover a new area of the search space ^[54].

The Chebyshev distance between two points x and y or two vectors with standard coordinates ${x}_{i}$ and ${y}_{i}$ is:

Eye closure ratio (ECR) is a scalar value that responds to the estimation of the eye closure state. ECR formula is indifferent to the direction and distance of the face and ensures the advantage of identifying faces from a distance ^[55,56]. Each eye is represented by six coordinates, as shown in Figure 7. ECR value is calculated by using the following equation:

Mouth opening ratio (MOR): Yawning is marked by mouth opening as shown in Figure 8. A parameter used to determine whether or not someone is yawning is the mouth-opening value. Like ECR, MOR is determined as:

If the eyes of a driver are closed and/or he is yawning, a drowsiness state is detected.

Drowsiness state: The following conditions must be crossed to detect a drowsy driver:

1) The driver is drowsy when the output of the detector module is greater than a drowsiness threshold. This value ranges generally between 0 and 1. In our case and after several tests, we have chosen a threshold of 0.3.

2) The average blink duration of a person is between 0.1 and 0.4 seconds. Thus, a person must be drowsy when his ECR is beyond this interval. Drowsiness is detected if the eyes are closed for over five seconds.

3) If the MOR is maximum, the driver is yawning and therefore he is drowsy.

4) The driver is obviously drowsy if he meets condition 1 and condition 3 at the same time.

3.3.2.   The prediction module

Sometimes, some falsely detected drowsiness situations causing false alerts can be recorded. These false alerts can annoy drivers. That is why we perform the prediction module, as shown in Figure 9, to alert only the real drowsiness states.

If a driver is detected drowsy, a captured image of the current frame passes through the three transfer learning CNN models to assess if it is a real drowsiness state or a false one. In order to achieve a performant DDD system, the three models' predictions will be combined to predict the ac-curate one based on an ensemble learning method. Ensemble learning is a machine learning tool where several models (VGG16, VGG19 and ResNet50) were trained to solve the same problem in order to boost the overall accuracy. For the proposed DDD system, we adopt the bagging fusion strategy as an ensemble learning method. The bagging combines all the predictions of all learners by majority voting and the most-voted class is decided in order to assess the accurate class of the driver (drowsy or not). If the driver is confirmed drowsy, then, the alarm is triggered.

To measure the effectiveness of the proposed approach, we need to evaluate the used CNN models. The evaluation metrics provide convenient measures of performance for the proposed system.

3.4.   Model evaluation metrics

The performance metrics that have been used to measure the ability of the different used models to detect drowsy subjects are explained below.

3.4.1.   Confusion matrix

A confusion matrix is a table arrangement that summarizes the prediction results on a classification task. Each line of the table represents the outcomes in an observed class while each column represents the outcomes in a predicted class, or vice versa.

There are four potential values to get in a classification task that can be defined as follows ^[57]:

● True Positive (TP): The predicted model outcome was true, and the observed value was true.

● False Positive (FP): The predicted model outcome was true, but the observed was false.

● False Negative (FN): The model expected a false outcome, while the observed value was true.

● True Negative (TN): The model predicted outcome and the observed outcome were false.

Confusion matrices can be used to calculate performance metrics for classification models. The most commonly used metrics to judge model performance are accuracy, precision, recall, and F1 score.

3.4.2.   Accuracy

Accuracy gives the percentage of the total number of the correctly predicted values. It should be as high as possible. The accuracy calculating formula is:

3.4.3.   Precision

The precision is the rate of correctly defined positive values. Its formula can be written as:

This metric allows us to calculate the rate of the positive predictions which are actually positive.

3.4.4.   Recall

Recall (or sensitivity) is the measure of true positive over the count of actual positive outcomes. It represents the percentage of observed positive cases that are correctly identified. The formula for recall can be expressed as:

Using this formula, we can evaluate our model if it is able to identify the actual true result.

3.4.5.   F1 Score

The F1 score is the Harmonic Mean of precision and recall. This metric highlights these two factors. The formula for the F1 score can be expressed as:

3.4.6.   AUC-ROC curve

The ROC (Receiver Operating Characteristics) curve is a significant evaluation metric to measure any classification model's performance in DL. This curve plots two parameters:

● True Positive Rate (TPR), which is the recall rate.

● False Positive Rate (FPR) is defined as follows:

TPR and FPR represent the y-axis and the x-axis of the ROC curve. The region under the curve is known as the Area Under the Curve (AUC). The larger the area, the better the performance.

4.   Experimental results

As we mentioned before, our DDD system is built based on DNN. Moreover, we tried to achieve the training using the traditional CNN model. A performance comparative analysis of the CNN model with the three DNN models used in the learning module of the DDD system has been performed based on the previous detailed evaluation metrics.

To evaluate the proposed DDD system, we trained the previously described models using the YAWDD dataset ^[44]. A total of 80% of the dataset was used for training, and only 20% was used for testing. The dataset used in the training phase was different from the one used in the testing phase. Furthermore, data augmentation techniques are used to the training set in order to overcome image transformations and noise. We referred to geometric transformations as data augmentation tools which are: Zooming, flipping and rotation in order to generate new data. The generation of new data was done during the learning step. All the data moved using forward backward propagation. At each iteration, the image passed through the data augmentation layer before reaching the convolution layers of the DL model. Finally, the model improved the accuracy and minimized the error at each iteration.

The data augmentation layer randomly generated a combination of the real image and other different augmented images. It may generate the original image as it is, one of the random modifications, or a combination of 2 or more of them, that guarantees a minimum of multiplication of data by 5 during 20 passages (20 epochs). This action has increased the number of images of the training dataset four times higher. Increasing the diversity of the training set had made an imperative improvement in the DNN testing performances. Figure 10 shows an example of an augmented image.

Table 2 lists the number of layers for the various models that are used.

These models were developed in open-source language Python using Collab API with all supporting libraries related to computer vision and deep-learning architectures as OpenCV, Keras, and Tensorflow tools on a PC with the following configuration: Intel® Core (TM) 10th generation CPU, 8 Go of RAM, Winodws 10, 64 bits and a Web Camera. The total epochs vary from 8 to 50 according to the model. The time processing is therefore different for each model. It increases unless the number of layer increases. However, on average, the DDD system took 0.22 seconds to train a single image for each model.

Figure 11 presents the confusion matrices for the different used models. This figure gives an overview of the attained accuracies.

According to the confusion matrices, the system is able to eliminate all the false positive values. This can consolidate the challenges raised in this work to conceive a reliable system. Table 3 depicts the performance metrics for the CNN model with the different DNN models.

Table 3 reveals that the ResNet50 model achieved the highest values for all metrics. The time processing is as higher as the number of layers increased and it is relative to all hardware and software materials. On other hand, the CNN model gives the lowest values at all. Transfer learning is therefore more suitable to solve the target task.

According to the achieved results, the ResNet50 model is the most efficient DNN model for the drowsiness state classification with a testing accuracy of 98.4%. Furthermore, VGG16 and VGG19 also achieved competitive results, as shown in the table.

The ROC curves corresponding to the DNN models used in our DDD system are presented in Figure 12.

The area under the ROC curves of all models confirms that all DNN models are good classifiers. Indeed, all the models present good AUC values, so they have a great ability to recognize the drowsiness state. The ResNet50 model is the most performant model to recognize the drowsiness state.

5.   Comparison and discussion

Several DDD systems were proposed in the literature. They used many and diverse methods and techniques to conceive their detection procedure. The most popular DDD system is the behavioral parameter-based techniques or also known as imaged based system. This system continuously monitors the drivers' physical behavior especially the facial expressions like eye closure ratio, eye blinking and yawning.

The comparative analysis of the proposed DDD system with such techniques can be performed by comparing the performance's metrics of the mentioned DDD systems in the paper or through other DDD systems that had used the same dataset utilized by our system (the YAWDD dataset).

Table 4 summarizes the DDD systems mentioned in the literature review of this paper with the one proposed.

According to Table 4, the best accuracy is assigned to our DDD system proposed in this paper.

Table 5 makes a comparison in terms of accuracies between DDD systems that used the same dataset utilized in this paper.

According to Table 5, all the systems used cameras or images to detect the fatigue and the drowsiness of the driver, and they show valuable accuracies. Besides, the proposed approach overshoots the other approaches that used the YAWDD dataset in terms of accuracy.

To conclude, experiments demonstrate that using car camera and DL technology simultaneously is promoting drowsiness detection. Indeed, the valuable information issued from the car camera can be boosted by DL technologies which capture various observable fatigue and drowsiness characteristics. The experiments also reveal that ensemble learning approaches can significantly increase the DDD system performance that will effectively promote robustness and reliability. Liu et al. ^[61] confirmed the same idea in their review paper.

The main advantage of the proposed DDD system is that it can be easily industrialized because it is cost-efficient, easy to use, non-invasive system and automatic. However, this system is very dependent on the quality of the image processing which makes the detection of the driver's state highly related to the quality of the image. Moreover, some conditions may affect the DDD systems like wearing sunglasses, sudden changes in lighting, and camera distance from the face. All these factors can reduce the accuracy or even lead to false detection. The major challenge for the DDD system is guaranteeing a high-quality camera and carefully considering environmental factors of the car during system development and testing.

Successfully, our DDD system is an advanced and intelligent system, comparable to other cutting-edge technologies like the Traffic Sign Recognition System (TSRS) ^[62]. By integrating the DDD system into the Advanced Driver Assistance Systems (ADAS) and/or Automated Driving Systems (ADS) in smart vehicles, we aim to provide valuable assistance to drivers in order to improve their efficiency and safety.

6.   Conclusions

Drivers' behavior, especially drowsiness, is the major cause of road accidents worldwide. Detecting and modeling the drowsiness state by means of DDD systems makes it possible to alert the driver about a dangerous situation. However, these systems have not yet been operated for many reasons especially inaccessibility and lack of performance. Therefore, it is necessary to build a reliable drowsiness detection system that can correctly and effectively detect real-time drivers' behavior. Based on the eye closure and mouth opening, we can perform a functional DDD system. In this study, we have proposed a hybrid DDD system to detect a drowsy driver in a real-time state. The working process has been divided into an offline process dedicated to the learning module and an online process for the detection and the prediction modules. For the offline training, we have applied data augmentation techniques for the used database to enhance the data. Besides, the pre-trained CNN models used in the learning module have achieved good performances for the classification of the driver's state and the recognition of a drowsy driver in the prediction module. In the online process, we have used transfer learning techniques to evaluate the drowsiness state. The considered system can eliminate false drowsiness alerts and to identify only the real ones. This hybrid approach increases the system performances which are consequently reflected by the obtained results. In the proposed study, one camera could be insufficient in case of fast head rotation, which leads to the vanishing of an eye from the captured image. This can be solved by using extra cameras in the car. Moreover, our approach studies only the features of a driver's face. It does not extract the features from limb movements. Future works should further consider other body parts as indicators.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This research has been funded by Deputy for Research & Innovation, Ministry of Education through Initiative of Institutional Funding at University of Ha'il – Saudi Arabia through project number IFP-22 122

Conflict of interest

All the authors declare that there are no potential conflicts of interest and approval of the submission.