Intelligent synthesis of hyperspectral images from arbitrary web cameras in latent sparse space reconstruction

Abbreviation: ACC: Accelerometer; ANN: Artificial neural network; BP: Blood pressure; BVP: Blood volume pulse; CNN: Convolutional neural network; DNN: Deep neural network; ECG: Electrocardiograph; EDA: Electrodermal activity; EEG: Electroencephalography; EMG: Electromyogram; GSR: Galvanic skin response; HR: Heart rate; HRV: Heart rate variability; IBI: Inter-beat interval; KNN: K-nearest neighbor: LASSO: Least absolute shrinkage selection operator; PCA: Principal component analysis; PPG: Photoplethysmography; RF: Random forest; RR: Respiration rate; RBF: Radial basis function kernel classifier; ST: Skin temperature; SVM: Support vector machine

1.   Introduction

Mental health problems are also known as mental health disorders, and they have become an important factor affecting individual growth and social stability ^[1]. According to the "Global Burden of Disease Study", the global prevalence of mental health disorders has always been above 10%, and about 450 million people have mental health disorders ^[2,3]. Long-term and persistent mental health problems increase the risk of cardiovascular disease, which is the leading cause of increased mortality worldwide ^[4]. Mental health disorders not only damage the physical and mental health of individuals, but they also increase the overall medical burden on society ^[5]. It is predicted that the total global medical expenditure for mental health problems will reach 16 trillion US dollars by 2030 ^[6]. Therefore, it is imperative to detect the mental health condition of individuals at an early stage and prevent the occurrence of mental health problems.

Smart sensors can accurately detect subtle changes (such as insomnia, headache, rapid heartbeat and muscle tension) in the body caused by stress in order to prevent and reduce the adverse effects of mental health problems ^[7,8]. With the increasing need for mental health care, the application of portable smart devices for monitoring mental health has received widespread attention and has been welcomed by most people ^[9]. At present, there are many studies on the use of wearable devices for mental health monitoring. Previous reviews have integrated the use of wearable sensors to detect stress levels, and there is a lack of exhaustive reviews on detection topics such as mental health problems or mental illness ^{[8,10,11,12,13]}. However, although some of the studies covered comprehensive information, the classification of detected signals were not described in detail in these reviews ^{[14,15,16,17]}. To fill this gap, this paper analyzes and integrates a large number of studies, covering the types of equipment used for data collection, common locations of sensors, common signals, advantages and disadvantages of technology, types of mental health states, etc. In this section, we discuss some existing studies. Abbreviations of nomenclatures used in this review paper are enumerated at the beginning of the article.

First, smart wearable devices have been widely used in the study of classifying and identifying individuals' daily stress and mental fatigue. Betti et al. ^[18] measured the electroencephalography (EEG), electrocardiography (ECG) and electrodermal activity (EDA) of healthy adults using wearable devices and a support vector machine (SVM) classification algorithm for statistical analysis; the results show that the accuracy of the classification algorithm based on salient features can reach 86% for detecting individual stress levels. Goumopoulos et al. ^[19] proposed a method to detect mental fatigue; they used a wearable device to collect the heart rate variability (HRV) of healthy subjects, and then selected an appropriate machine learning model to predict mental fatigue with high accuracy. Additionally, the role of wearable devices in the prediction and classification of mental health assessment and emotional states cannot be ignored. Sano et al. ^[20] explored the relationships between the academic performance, sleep quality, stress perception and mental health of college students using wearable devices and smartphones; they found that college students' academic performance, sleep quality and stress perception can significantly predict their mental health level, and that the accuracy rate of the algorithm is as high as 92%. Costa et al. ^[21] found that wearable devices can acquire and analyze social signals and identify the current negative emotions of individuals, and that, when it identifies a negative emotion, the device activates the emotional regulation system to help the individual stabilize their emotions. Overall, wearable devices have been widely used in the field of mental health with strong practicability. The main contributions of this paper are given as follows:

1) Recent artificial intelligence (AI)-based wearable devices methods and techniques are reviewed in this paper. This study comprehensively explores the application of AI-based wearable devices in the field of mental health, focusing on stress levels, mental health problems and mental illness.

2) This study covers comprehensive and detailed information, including types and application mechanisms for AI-based wearable devices, common locations of sensors, common signals, the advantages and disadvantages of technology and types of mental health states.

3) The articles included in this paper are from 2017 to 2022, and the articles are relatively new and cutting-edge.

The remainder of this paper is organized as follows. Section 2 introduces the review of body signals and mental health. Section 3 summarizes the methodology and Section 4 gives the literature survey. Finally, application, discussion and conclusion are given in Sections 5, 6 and 7.

2.   Body signals and mental health

The autonomic nervous system has two branches: the sympathetic nervous system (SNS) and parasympathetic nervous system (PNS). The SNS can rapidly stimulate some vital organs (such as the heart, lungs, and sweat glands), and the PNS can inhibit this excitation; they complement the regulation of important bodily functions and maintenance of physiological homeostasis ^[22]. When an individual is stressed, the individual will activate the protective mechanism; the SNS will be activated, the heart rate will increase, the breathing will increase, sweat will be secreted and other physiological systems will also be affected ^[23,24]. Additionally, the hypothalamus axis is a hormonal system that is initiated in response to stressors and drives a cascade of endocrine responses, culminating in the secretion of cortisol ^[22]. Therefore, individual health, especially mental health, can be known by observing the activity of the SNS and hypothalamus axis. Finally, language signals can also reflect an individual's mental health condition. Language is the basis for inferring individual thought processes; it provides a special perspective for thinking observations ^[25]. When an individual is in a state of tension, the respiratory rate increases and the glottis pressure increases, prompting an increase in the pitch of the vocalized part ^[25]. Therefore, clinicians mainly diagnose and treat mental illness through language and pronunciation ^[26]. As mentioned above, subtle changes in the body caused by stress can be detected by sensors connected to the body and hardware devices to accurately predict mental illness ^[27]. Therefore, AI-based wearable devices can collect one or more signals of the individual to identify and analyze the mental health level.

3.   Methodology

3.1.   Search strategy

This review incorporated searches for relevant papers in the databases of Web of Science, Embase, MEDLINE and Engineering Village. The selected papers ranged from 2017 and 2022, which covers the recent developments in this area. The basic sets of keywords were "Psychological assessment", "psychological prediction", "psychological monitoring", "mental health", "psychological health", "mental illness", "psychological disorders" and "wearable device". A total of 1537 papers were retrieved by using all keywords, and 25 papers were finally included. The search methodology for this article was designed to provide a detailed review of published research papers on wearable sensor-based mental health detection. Figure 1 shows a flowchart describing the search strategy used for this review article.

Figure 1.  Flow diagram of the systematic search strategy.

DownLoad: Full-Size Img PowerPoint

3.2.   Inclusion criteria

1) The date of publication of the selected papers was between 2017 and 2022. 2) Conference papers and journal papers were included. 3) The research direction for the paper was mental health detection (focusing on stress, mental health problems, mental illness, etc.) and AI-based wearable devices for data collection.

3.3.   Exclusion criteria

1) Research published in a language other than English was excluded. 2) Studies structured as editorials, press releases, research highlights or letters, review articles, reviews and technical papers were excluded. 3) Duplicate papers, related review papers and papers not related to the topic were excluded.

4.   Literature survey

4.1.   Wearable devices for mental health monitoring

We use smart/embedded wearable devices, such as smartwatches, that collect signals from the body ^[28]. This data can be processed by sensors on smart/embedded devices. WiFi is used to send the collected data to cloud servers ^[7]. In cloud computing, advanced AI models can be applied to mental health monitoring big data to generate simple knowledge through the use of expensive processing and advanced AI algorithms ^[29], such as Welch's algorithm ^[30], SVM algorithms ^[31] and period graphs ^[32], for the analysis of data. Using these AI algorithms, then by analyzing the information, we will be able to extract sensor data with the appropriate classification characteristics and then classify their mental health states ^[33]. Artificial intelligence algorithms such as information and SVMs are used again. The main function of an analog classifier is to classify different classifiers ^[34]. By ensuring the rapidity of calculation, feature patterns are transformed into related classes and the classes most suitable for a single feature are predicted. During the classification process, each feature will be analyzed to find the most suitable feature ^[35]. They will be analyzed separately to predict stresses. Once the classification is complete, the smart device displays their mental health rating on the screen. Therefore, people can improve their mental health through self-management (as shown in Figure 2). The virtual reality-augmented reality (VR-AR) based digital twin is the critical element for the intelligent development of AI-based wearable devices ^[36]. Finally, all of the elements are integrated into a smart device's ecology for the future direction framework.

Figure 2.  Intelligent ecology flowchart of AI-based wearable devices with data storage, transaction, interaction and communication networks.

DownLoad: Full-Size Img PowerPoint

Mental health detection systems based on wearable devices mainly consist of sensors, data acquisition, data pre-processing, feature extraction, machine learning algorithms and mental health condition classification. The general scheme of mental health prediction is shown in Figure 3. Section 4 will systematically introduce the current related research work according to the workflow of the wearable mental health detection system, as well as classify, compare and statistically analyze the existing research results.

Figure 3.  General scheme for the analysis of mental health.

DownLoad: Full-Size Img PowerPoint

4.2.   Data acquisition devices

Currently, the commonly used signals in mental health detection are HRV, EEG, galvanic skin response (GSR), skin temperature (ST), blood pressure (BP), sleep, salivary cortisol and blood oxygen saturation (SpO2). In most studies, researchers use multiple indicators to accurately reflect an individual's mental health. At the same time, the wearable device can also collect indicators such as steps, activity category, posture and activity trajectory. According to the needs of users and the functions achieved by the device, the device will be worn on different parts of the body. Common wearing positions are shown in Figure 4.

Figure 4.  Common positions of AI-based wearable devices.

DownLoad: Full-Size Img PowerPoint

4.3.   Pre-processing and feature extraction

To solve the noise phenomenon in the data acquisition module, data pre-processing is required to reduce the impact of noise on the performance of mental health detection. Common filtering algorithms in mental health detection systems are wavelet transforms, the Kalman filter, etc. These filters can filter out high-frequency noise signals in the data ^[37,38].

The pre-processed data need to extract corresponding features for training the mental health detection model. Among them, the common feature extraction method is to use the sliding window algorithm to extract the features in the corresponding window, and then put the trained features into the trainer for training to form a mental health detection model. Table 1 summarizes the features extracted by the feature extraction module of an existing detection system. The commonly used feature values for feature extraction are Mean, Standard Deviation (StDev), Skew, Root Mean Square (RMS), Kurtosis, Quart1, Quart3, Inter-Quartile Range (IRQ), Sum, Frequency, etc.

Table 1.  Statistical features commonly used in mental health.

References	Feature	Description
[19,39,40,41,42,43,44,45,46,47,48,49,50,51]	Mean	$Mean = \sum\limits_i {s_i^2} /n$
[19,39,40,41,42,43,44,45,46,47,49,50,51]	StDev	$sd=\sqrt{\sum\limits _{i=1}^{N}{\left({x}_{i}-\mu \right)}^{2}/(N-1)}$
[40,45]	Skew	${G}_{1}=\frac{n}{(n-1)(n-2)}\sum \limits_{i=1}^{n}{\left(\frac{{x}_{i}-\overline{x}}{s}\right)}^{3}$
[19,40,43]	RMS	$rm\text{Δ}(s)=\sqrt{\sum \limits_{i}{s}_{j}^{2}/x}$
[39,40,45,51]	Kurtosis	$K = \mu /\sigma$
[39,40,45]	Quart1	Median of lower 25% of the values
[39,40,45]	Quart3	Median of up 25% of the values
[39,40,45,51]	IRQ	Difference between Quart1 and Quart3
[41,42,44]	Sum	$S = \sum\limits_{i = 1}^N {{w_i}}$
[45]	Variance	$s{d}^{2}=1/n\sum \limits_{i=1}^{n}{\left({x}_{i}-\mu \right)}^{2}$
[43]	Coefficient of variation	$CV = sd/\mu$
[45]	Inverse coefficient of variation	$ICV = \mu /sd$
[18,19,41,43,46,47,50,51,52]	Frequency	Peak frequency, low-frequency power (LF), high-frequency power (HF), very-low-frequency power (VLF), prevalent low frequency (pLF), prevalent high frequency (pHF), the ratio of LF to HF (LF/HF), (From Lomb–Scargle) LF, HF and LF/HF

---

 | Show Table

DownLoad: CSV

4.4.   Machine learning techniques

Machine learning technology refers to the method of using some special formulas to guide the computer to use the known data to build a model that can make predictions for new situations. Common machine learning algorithms for mental health detection include SVMs ^{[18,19,39,43,44,48,51,52,53,54,55]} k-nearest neighbor (KNN) algorithms ^{[19,48,51,53]}, random forests (RF) ^[40,45,51], artificial neural networks (ANNs) ^[39], logistic regression (LR) ^{[19,42,56,50,51]}, decision trees (DTs), Bayesian networks (BNs) ^[48], linear discriminant analysis (LDA) ^[51] and principal component analysis (PCA) ^[47,51]. The SVM is the most used algorithm for mental health detection among them. This paper summarizes the artificial intelligence algorithms involved in mental health detection and briefly describes their definitions, advantages and disadvantages, as shown in Table 2.

Table 2.  Comparison analysis summary of several existing machine learning techniques.

Machine learning algorithms	Definition	Advantages	Disadvantages
Support Vector Machine (SVM)	In order to better find the hyperplane that can divide the feature space, the low-dimensional data are mapped to the high-dimensional plane	Works well with any complex problem, can scale high-dimensional data, risk of overfitting is lower; It shows high robustness and strong generation ability with low overfitting	Sensitive to missing data; It is difficult to implement large-scale training samples; There are difficulties in solving the multi-classification problem
K-Nearest Neighbor (KNN)	In the feature space, if k samples near a sample belong to a category, that sample also belongs to that category	Simple thought, mature theory; Can be used for multiple classification, high accuracy, no assumptions on the data, not sensitive to outliers	Poor generalization ability and slow prediction speed
Random Forests (RF)	A classifier for multiple decision trees	It can handle high-dimension data, has strong model generalization ability, and is fast in training. It can balance errors in unbalanced data sets, maintain accuracy if a large part of features is lost	In noisy classification or regression problems, overfitting and attribute weights produced based on data of different attribute values are not credible
Artificial Neural Network (ANN)	A mathematical model for distributed parallel information processing that mimics the features of human neural network	It has a high accuracy of classification, strong parallel distribution processing ability, strong distribution storage and learning ability, adaptability, strong robustness and fault tolerance to noise nerve	Many parameters are required, which are not suitable for small samples, high calculation cost, long learning process, and intermediate results cannot be observed
Logical Regression (LR)	The result of linear regression is mapped between 0 and 1 by a sigmoid function. The result of the mapping is the probability that the data sample point belongs to a certain category	The computational cost is low, the implementation is easy to understand, the robustness to small noise in the data is good, and it is not particularly affected by slight multicollinearity	It is easy to underfit, and the performance effect is not good when the classification accuracy is not high, the data features are missing or the feature space is large
Decision Tree (DT)	A prediction model is a mapping between object attributes and object values	Easy to understand and explain, visualization of tree structure, less data required for training, fast analysis speed, and strong ability to process samples with missing attributes	Overfitting, poor generalization ability and unstable calculation results
Bayesian Network (BN)	Graphical networks based on probabilistic inference	The process is simple and fast, efficient for multi-classification problems, and requires a smaller sample size	Sample attributes do not work well when they are associated and are sensitive to input forms
Linear Discriminant Analysis (LDA)	The labeled data is projected to a lower dimensional space so that the points after projection are distinguished by category. Points of the same category are closer to each other in the space after projection, and the differences are as far as possible	It can reflect the differences between samples and use the prior knowledge experience of categories in dimension reduction	Easy to fit, restricted by sample type
Principal Component Analysis (PCA)	A method of multivariate statistical analysis in which several variables are transformed linearly to select a smaller number of important variables	Eliminate the mutual influence between indicators, small workload	The amount of information must be kept at a high level after dimensionality reduction, and the interpretation of principal components is usually fuzzy

---

 | Show Table

DownLoad: CSV

5.   Application

Mental health problems are often accompanied by obvious body responses, and wearable sensors play an important role in mental health and its related applications. They can collect an individual's EEG, heart rate (HR), HRV, GSR, BP, body temperature, respiratory rate, etc., to monitor mental health. This section summarizes and discusses the signals commonly used in mental health detection, and briefly summarizes the relevant research on AI-based wearable devices in mental health detection over the past five years. The main contents include mental health disease types, parameters, results, etc., as shown in Table 3.

Table 3.  Mental health applied research model and summary of results.

Study	Type	Device	Signals	Model	Result
Can et al., 2019 [51]	Stress	Samsung Gear S, S2 and Empatica E4 wristband	End Diastolic Velocity, HRV and ACC	PCA, LDA, SVM, KNN, RF and LR	The accuracy of the multi-layer perceptron algorithm is 92.19%.
Betti et al., 2018 [18]	Stress	Mobile EEG headset, Chest belt	HRV, EDA, EEG and Salivary cortisol	SVM	The classification algorithm based on salient features has an accuracy of 86%.
Egilmez et al., 2017 [40]	Stress	LG smartwatch	GSR	RF	The pressure detection F-value of the system is 88.8%.
Gjoreski et al., 2019 [39]	Stress	Empatica device's wristband	HR, BVP, IBI, ST and ACC	SVM	Experiments with 55 days of real data showed that the method detected stress events with 95% accuracy.
Ahn et al., 2019 [52]	Stress	Head-mounted electrode	ECG, EEG	SVM	The accuracy of the classification of pressure states using SVM technology and the 5-fold cross validation of EEG and HRV features was 87.5%.
Wu et al., 2019 [50]	Stress	Smart shirt	Cortisol, HRV	LR	Mental stress index is sensitive to acute stress and can predict the level of association between normal individuals and the stress group with about 97% accuracy.
Jesmin et al., 2020 [53]	Stress	Smartwatch	HRV, GSR and SpO2	SVM, KNN	Multi-sensor data fusion can better monitor individual stress levels.
Bobade et al., 2020 [41]	Stress	Chest-worn device, RespiBAN Professional and a wrist-worn device Empatica E4	ACC, ECG, BVP, ST, RR, EMG and EDA	Machine Learning	Using machine learning technology, the accuracy of Class Ⅲ and Class Ⅱ classification problems reached 81.65 and 93.20%, respectively, and using deep learning technology, the accuracy of Class Ⅲ and Class Ⅱ classification problems reached 84.32 and 95.21%, respectively.
Silva et al., 2020 [49]	Stress	Smartwatch	HRV	ANN	Several parameters of the HRV under pressure were significantly different from baseline.
Kim et al., 2020 [42]	Stress	Empatica E4 wristband	GSR	LR	The overall classification accuracy of the model was 85.3%, and the cross-validation accuracy was 83.2%.
Han et al., 2020 [48]	Stress	Smart wrist band, Empatica E4 wristband and a Shimmer ECG device	ECG, PPG and GSR	KNN, SVM, and BN	The accuracy of 10-fold cross-validation was 94.55%, while the accuracy of subject cross-validation was 85.71%. In daily settings, the system had 81.82% accuracy in assessing stress.
Tonacci et al., 2019 [62]	Stress	Shimmer™, Shimmer3™	GSR, ECG	Pan Tompkins algorithm	HR increased with both sensory (olfactory) and cognitive stimulation, and there was a similar decrease in HRV.
Tiwari et al., 2019 [43]	Stress and anxiety	OMsignal smart-shirt and a Fitbit Charge 2 smart-bracelet	HRV	SVM	Overall, HRV characteristics calculated from wearable vest data (with higher temporal resolution) proved to be more effective in predicting stress and anxiety.
Patlar Akbulut et al., 2020 [47]	Stress and anxiety	Wrist-worn wearable device	ECG, GSR, ST, SpO2, Blood sugar and BP	PCA, Feed-forward neural network	Method had an average accuracy of 92 and 89% in distinguishing the stress levels of the experimental group and the control group, respectively.
Sano et al., 2018 [54]	Stress and mental health condition	Q-sensor and Motion logger	SC, ST, ACC and Motion logger	SVM, LASSO and RBF	The sensors were 78.3% accurate in classifying students as high or low stress groups, and 87% accurate in classifying students as high or low mental health groups.
Fraiwan et al., 2018 [65]	Mental health condition	Smart glove	ST and HRV	HRV algorithm	There was significant variability in skin resistance and HRV under relaxation and stimulation.
Chen et al., 2018 [68]	Mental health condition	Smartwatch	Speech	Transfer learning	The use of wearable long-term speech-to-social data sets to categorize various sound scenes in unconstrained and natural environments yielded good results.
Wen et al., 2020 [44]	Anxiety	The MP150 multi-channel physiological recorder	HR (ECG electrodes)	SVM	The classifier achieved 81.82% accuracy in detecting a high anxiety state.
O'Brien et al., 2017[56]	Depression	Wrist-mounted device	ACC	LR	General physical activity was significantly reduced in depressed older adults compared with healthy controls.
Narziev et al., 2020 [67]	Depression	Smartwatch	Motion logger and HR	Machine learning	Participants' ecological transient assessment self-report and passive perception (sensor data) were highly correlated between physical activity, mood and sleep levels; Depression detector verified the feasibility of grouping and classification with an accuracy rate of 96.00% (StDev = 2.76).
Zanella-Calzada et al., 2019 [45]	Depression	Actiwatch	Motion Logger	RF	Patients with depression were accurately detected in 86.7% of the cases, while those without depression were found in 91.9% of the cases.
Gentili et al., 2017 [55]	Bipolar disorder	Sensorized t-shirt	HRV, respiration and ACC	SVM	After comparing the HRV features of the emotional states before and after, the classification accuracy of the SVM in the target state was significantly improved.
Zheng et al., 2019 [46]	Emotion recognition	Head/wearable eye-tracking glasses, EmotionMeter hardware	EEG and Eye-movement signal	Multi-mode deep neural network model	The modal fusion performance of the multi-mode deep neural network was significantly improved, and the average accuracy of the four emotions (happiness, sadness, fear and neutral) reached 85.11%.
Goumopoulos et al., 2022 [19]	Mental fatigue	Zephyr HXM-BT, Chest-strap built-in ECG sensor	HRV	SVM, KNN and LR	Mental fatigue caused by cognitive load can be detected by wearable commercial devices and a single biomarker.
Qiu et al., 2020 [38]	Mental fatigue	Head-mounted devices	ECG, EEG and ST	Wavelet transform, Kalman filtering	Through the fatigue evaluation experiment of 10 healthy subjects, it was verified that the device can reliably detect the mental fatigue state by monitoring and analyzing EEG, ECG and proximal ST data.

---

 | Show Table

DownLoad: CSV

5.1.   Mental health detection using EEG

EEG is a detection method that records the electrical activity of the cerebral cortex by connecting multiple electrodes to the scalp ^[57] to reflect the emotional changes of individuals to specific stimuli. The stress state of the human body is mainly identified by the changes in the alpha and delta bands of the EEG. When people are under high levels of stress, alpha decreases and delta increases in the EEG, whereas, when they are relaxed or inactive, alpha increases and delta decreases ^[58]. Ahn et al. ^[52] used a binaural hanging device that combined EEG and ECG signals and found that the accuracy of EEG and ECG signals in continuous stress assessment was 87.5%. Additionally, based on the multi-modal emotion recognition framework of EmotionMeter, some scholars found that the individual's happiness, sadness, fear and neutral emotions can be distinguished and identified by EEG signals ^[46]. Finally, EEG combined with ECG signals, body temperature and other data can reliably detect the mental fatigue state of an individual, which provides support for the popularization and specific implementation of mental health assessments ^[38]. Table 3 shows the studies and their details on the use of EEG for mental health detection. EEG is one of the most effective physiological signals for detecting psychological stress, and the combination of ECG signals and machine learning applications can improve the accuracy of psychological stress detection.

5.2.   Mental health detection using ECG

Stressors induce changes in myocardial activity, promote blood distribution to vital organs and cause changes in the HR and HRV ^[59]. ECG is a reliable technique that uses electrodes attached to the body to record the electrical potential produced by heart muscle cells to observe changes in HRV ^[59]. Studies have shown that HR and HRV signals are closely related to mental health problems (such as anxiety and depression) ^[60], and they have an important role in mental health detection. Tiwari et al. ^[43] found that HRV data obtained through smart t-shirts and smart bracelets can more accurately assess the degree of anxiety and depression of individuals. Second, HRV characteristics are closely associated with an individual's clinical course and mood changes. Gentili et al. ^[55] showed that the HRV characteristics (including time domain, frequency domain, nonlinearity) of patients with bipolar disorder can significantly predict their clinical course and emotional changes. Additionally, Akbulut et al. ^[47] believe that the stress state of an individual can be accurately identified by signals such as the HRV obtained by wearable devices. Table 3 presents an overview and the details of mental health testing using ECG.

5.3.   Mental health detection using GSR

Sweat is a biological fluid produced by sweat glands. Sweat glands are distributed almost all over the body, with high-density distribution in the palms, soles and forehead; they are only stimulated by the sympathetic nerves, which are activated under tension ^[14], so the level of sweat secretion is the body's response to stress. Since sweat contains ions, small molecules and even macromolecular substances such as proteins, it has electrical conductivity ^[61]. Therefore, the EDA signal is obtained by connecting two electrodes on the skin surface, and the voltage between them is measured to monitor the stress. Sriamprakash et al. ^[33] collected the GSR, HRV and other indicators, extracted individual stress characteristics by using Welch's algorithm and periodograms, etc., and used SVM and KNN algorithms to classify levels of mental health. Tonacci et al. ^[62] captured the ECG and skin conductance (SC) signals of subjects by using wearable sensors and compared the effects of cognitive stress and olfactory stimuli on human mental health. Additionally, EDA and GSR also change with different emotions; this can effectively reflect the state of emotional calm or emotional arousal ^[63]. Evidence shows that the emotional state of patients with metabolic syndrome can be analyzed by using indicators such as the GSR, HRV and body temperature detected by wearable devices ^[47]. The above studies have confirmed that combining GSR with HRV, body temperature and other parameters can improve the prediction rate for individual stress, and accurately provide and detect information about individual emotional states.

5.4.   Mental health detection using ST

High levels of stress can cause muscle tension, increase the HR and increase the work of the vital organs; additionally, the blood will be shunted from the limbs to directly act on the vital organs. In just a few minutes, the body surface temperature can change ^[64]. Most of the studies involved the collection of body temperature data to detect mental health conditions. Fraiwan et al. ^[65] designed a mobile mental health monitoring system for continuous monitoring of changes in ST, skin electrical impedance and HRV; they found that, when the body was exposed to relaxation or stimulation, the skin electrical resistance value and HRV exhibited significant variability. Engelniederhammer et al. ^[66] monitored the EDA and ST in subjects wearing a smart wristband (Bodymonitor); the results showed that being in a crowded environment triggers aversive emotional responses (i.e., stress, anger and fear) in subjects. Sano et al. ^[54] had 201 college students wear two sensors on each wrist: a Q-sensor (for measuring SC, ST, triaxial acceleration (ACC)) and motion recorder (for measuring acceleration and ambient light data); the study showed that wearable sensor eigenvalues are more accurate than cell phones for classifying poor mental health (87%) and stress (78.3%) when using machine learning techniques. In conclusion, the ST, combined with parameters such as EDA, is more accurate for detecting emotional changes and stress levels.

5.5.   Mental health detection using speech and behavior

With the development of artificial intelligence, the natural language processing program of the computer can more accurately identify the subtle barriers in language before the onset of mental illness, achieving the effect of accurately predicting the mental illness ^[27]. Corcoran et al. ^[27] confirmed that the automatic learning of speech classifiers with reduced semantic consistency, increased semantic consistency differences and reduced use of case pronouns can predict psychotic episodes with an accuracy rate of 83%. Language signals help to improve the accuracy of mental disease prediction and have become a powerful tool for psychiatric diagnosis and treatment.

In addition, mental illness is associated with changes in an individual's behavior, especially physical movement and social habits. AI-based wearable devices (accelerometers, global position system sensors) can detect these abnormal behaviors and effectively reduce the incidence of individual mental illness. Zanella-Calzada et al. ^[45] obtained data on motor activity recorded by wearable devices from a depression database, extracted the statistical features and then fed them to an RF classifier to detect subjects with depression; the results showed that those subjects with the presence of depression had an 86.7% likelihood of being correctly classified, while the specificity showed that those subjects with the absence of depression had a 91.9% likelihood of being classified with a correct response. Additionally, Narziev et al. ^[67] recruited 20 participants from four different levels of depression groups and established a machine learning model to automatically classify the categories of depression in a short period of time; the results of this pilot study revealed high correlations between participants' ecological momentary assessment self-reports and passive sensing (sensor data) in terms of physical activity, mood and sleep levels. All of the above studies have shown that the role of speech signals and behavioral signals in the detection of mental health states such as depression, mood disorders or schizophrenia cannot be ignored.

6.   Discussion

Multiple studies have integrated the use of wearable sensors to detect stress levels, and there is a lack of exhaustive reviews on topics such as mental health or illness detection. However, some reviews comprehensively cover information, but do not provide details such as machine learning methods, types of mental health states and the classification of detection signals. To fill this gap, this paper presents an extensive review of mental health detection using wearable sensors.

The significant observations from this review are listed below.

● Most wearable sensors are placed on the wrist (such as smartwatches and smart wristbands), chest (smart t-shirts, chest straps, etc.) and head (head-mounted devices, etc.) for data collection.

● The Empatica, EmotionMeter and SHIMMER platforms have been popularly used for data collection.

● HRV, EEG, GSR and ST are the most distinctive signals for detecting mental health conditions.

● SVM, LR and KNN have been widely used in the classification of mental health conditions.

● Most of the disease types studied focused on psychological stress levels and mental health problems (anxiety and depression).

The identified challenges are described below.

● Compliance: The collection of subjects' physiological indicators and daily activity data is affected by individual living habits, which makes the detection process more complicated and thus degrades the performance of the mental health detection system. Therefore, the mental health assessment based on wearable devices can only do a rough screening and cannot directly assess the disease accurately.

● Singleness of indicators: most studies involve the collection of only a single category of signals, such as physiology, speech and behavior, while fewer collect a combination. Since different categories of data and signals reflect different aspects of mental health, the integrated collection of all types of signals is essential for the accurate screening of mental health problems.

7.   Conclusions

Globally, mental health problems are still serious social public health problems, and mental health has always been detected via subjective and time-consuming methods. Wearable sensors have great potential as a continuous monitoring system for mental health. As the needs for mental health care services increase, so does the production of AI-based wearable devices. This paper reviewed the types of equipment, common locations of sensors, common signals, advantages and disadvantages of technology and types of mental health states in previous studies. First of all, the most common types of states in mental health testing are psychological stress and mental health problems (anxiety and depression), and most wearable sensors are placed on the wrist, chest or head. Additionally, it was found that there are many AI-based wearable devices on the market that mainly collect physiological signals for mental health detection, and that there are few studies on detecting mental health with speech signals and behavioral signals. Among the physiological signals, the HRV, EEG, ST and GSR signals are the most effective parameter indicators for detecting the mental health condition and this conclusion is helpful for researchers to create devices to improve people's mental health level. Finally, this paper integrated important information, such as the technical advantages and limitations of previous studies; it was found that the most common classifiers are SVM, KNN and LR algorithms, and that these algorithms provide the most reliable results for mental health detection. In the future, with the continuous innovation of science and technology, deterministic annealing neural networks ^[69,70], convolutional neural networks ^[71,72,73], the logarithmic descent direction algorithm ^[74], Laplacian-Hessian regularization semi-supervised extreme learning machine ^[75,76] and back-propagation neural networks ^[77] are also expected to be applied and explored in the field of mental health detection. In a word, the goal of mental health testing is to develop an efficient and affordable model with high accuracy. This review will help future researchers design optimal wearable devices to achieve their goals for mental health detection.

Acknowledgement

This work is supported by the Changsha Natural Science Foundation No. kq2202422 and the Chinese Scholarship Council (CSC) under grant No. 202006370101.

Conflict of interest

All authors declare no conflicts of interest in this paper.