Crossing time windows optimization based on mutual information for hybrid BCI

1.   Introduction

Brain-computer interface (BCI) is a significant way of direct communication between the human central nervous system and the computer. It means, without the use of limbs, sounds, or any movements that require muscle activity, BCI can analyze brain induction signals which are related to the expected tasks. So BCI can generate commands to control external devices ^[1,2,3]. BCI systems also consist of signal acquisition, pre-processing, feature extraction, classification, external communication with the device, and feedback stages.

Electroencephalogram (EEG) ^[4,5], Electrocardiogram (ECoG) ^[6], Functional near-infrared spectroscopy(fNIRS) ^[7,8,9], Functional magnetic resonance imaging(fMRI) ^[10,11] and Magnetoencephalography (MEG) ^[12] are common signal acquisition modality in BCI systems. EEG is a bioelectrical imaging method that uses scalp electrodes to measure the voltage fluctuation caused by the electrical activity of cortical neurons. EEG stands out because of its high time resolution, frequency-domain characteristics, convenience, and safety ^[13,14]. Due to volume conduction effect in the cerebral cortex, it is difficult to determine whether EEG signal is generated near the surface of the cerebral cortex or in deeper areas. The spatial resolution of EEG is relatively low and it is susceptible to electrical noise and motion artifacts ^[15,16]. Hence, the classification accuracy of EEG is seriously damaged. fNIRS is an optical imaging modality that evaluates the hemodynamic activity in the brain. fNIRS can be used to measure the concentration changes of deoxyhemoglobin (HBR) and oxyhemoglobin (HBO) ^{[17,18,19,20,21]}. fNIRS stands out because of its non-invasive, portability, safety, and relatively high spatial resolution ^[22,23]. However, due to the slow response of blood oxygen concentration, the temporal resolution of fNIRS is low.

In the past few years, researchers have proposed hybrid BCI systems to overcome the limitations of single-mode systems ^{[24,25,26,27]}. It is feasible and reliable to use EEG and fNIRS as hybrid BCI to detect physiological signals of the specific subject. And hybrid BCI is widely applied in many cognitive rehabilitation fields ^[28,29,30]. Since EEG and fNIRS are specific time domain information rich, many researchers analyze the time domain of EEG and fNIRS ^[31,32]. Previous approaches were to segment synchronous EEG and fNIRS data to analyze the physiological state of the brain in different periods of the task ^[33,34]. And different time windows of signal have a significant impact on classification performance ^[35]. However, it ignores that the specific modal signals have various physiological characteristics. There are different matching degrees of time-domain information between EEG and fNIRS in the acquisition process. EEG signals reflect electrical activity originating as a result of neuronal firing when each task or activity is performed ^[36]. On the other hand, hemodynamic activity appears in the form of blood flow changes that result from neuronal firing ^[37,38]. Therefore, the electrical signal represented by EEG and the blood oxygen signal represented by fNIRS will not be activated at the same time when the subjects are stimulated by the task.

To solve the above problems, a crossing time windows optimal method based on Fisher Lasso feature selection (FLFS) algorithm for hybrid BCI is proposed. Firstly, the proposed method consists of multi-time segmenting and temporal band-pass filtering. Secondly, EEG and fNIRS time windows are crossed and combined. Thirdly, FBCSP and statistical methods are used to extract EEG and fNIRS features respectively. To select optimal time window, all features in each time window are sorted based on mutual information (MI). Finally, the features are put into FLFS and LDA for feature selection and classification.

Parts of this paper are arranged as follows: The second section describes the preprocessing method; the third section describes the dataset and experimental results; the fourth section discusses the experimental results; and the last part summarizes this full paper.

2.   Methods

As shown in Figure 1, it is a crossing time optimal combination method based on sparse feature selection for hybrid BCI. This method mainly includes the multi-time segmenting of EEG and fNIRS data; then FBCSP and statistical method are the hybrid feature extraction method; Finally, feature selection and classification are executed.

2.1.   Data segmenting and band-pass filtering

In the first part, EEG raw signals are decomposed to a total of I time segments. The time segments are ${T}_{i} = \left[L\times i, L\times i+w\right], 0\le i\le I$, where the unit is second and $L$ denotes step size of the window, $w$ denotes length of the window. In the fNIRS system, the fluctuating concentration of HBO and HBR are converted from the measured light intensity by the modified Beer-Lambert law ^[39,40]. And they are not obvious in frequency but have good time characteristics. Therefore, this part only segments the time of the fNIRS signal, and the time segmenting is the same as EEG. ${F}_{s}\in {R}^{c\times k}$ means the fNIRS signal in the ${s}^{\text{th}}$ time segment, where k is the sampling point and $k = \mathrm{1, 2}, ..., K$, c represents the number of channels.

2.2.   Feature extraction

2.2.1.   EEG Feature extraction

CSP is a spatial filtering algorithm for two classes of tasks, which can extract the spatial distribution components of each class from multi-channel EEG data. ${X}_{1}$ and ${X}_{2}$ are signal matrices for two classes of tasks respectively ^[41]. The covariance of the signal matrix is calculated and normalized:

where $trace\left(\cdot \right)$ is the trace of the matrix. Then find the covariance matrix $R$ of the mixed space and decompose eigenvalue:

where ${\stackrel{-}{R}}_{i}$ is the covariance matrix of two classes of task and $i = \mathrm{1, 2}$. $U$ is eigenvectors of matrix R, $\mathit{\Lambda }$ is eigenvalue matrix. So the whitening matrix $P$ can be obtained:

Then matrix ${R}_{1}$ and ${R}_{2}$ whiten and used by principal component analysis(PCA):

where ${B}_{1}, {B}_{2}$ are the eigenvectors of a matrix ${S}_{1}$, ${S}_{2}$ and ${B}_{1} = {B}_{2}$. ${\mathit{\Lambda } }_{1}+{\mathit{\Lambda } }_{2} = I$, where $I$ is the identity matrix. The sum of the eigenvalues is 1. When the eigenvalue of one class is the largest and the other is the smallest. It is the reason that the optimal spatial filter $W$ can distinguish two-class samples.

where $W\in {R}^{c\times 2m}$. Finally, the feature vector $f$ with high ability of discrimination is obtained.

where ${f}_{ij}\in {R}^{1\times 2m}$ represents the feature vector extracted by CSP from the EEG signal in the ${j}^{\text{th}}$ frequency band and the ${i}^{\text{th}}$ time segment.

Since the performance of the spatial filter in the CSP algorithm depends on the frequency band of the EEG, FBCSP is to first pass the EEG signal through a bandpass filter to form multiple bands and then extract the CSP features from these sub-bands ^[42]. Therefore, the segmenting data is put into the third-order Butterworth band-pass filter to divide the multi-band.

2.2.2.   fNIRS feature extraction

The statistical features of fNIRS were employed by signal mean ^[43], signal slope ^[44], signal variance ^[45], signal peak ^[45,46], signal kurtosis and signal skewness. Signal mean and the signal peak will be applied in this part ^[47].

Signal mean

The average concentration of HBO and HBR is calculated according to the Eq 7:

where N is the number of sampling point and X is the HBO or HBR data. $M$ is the mean feature vector and s denotes the fNIRS data samples in the ${s}^{th}$ time segment.

Signal peak

The signal peak is the maximum value of the signal in the time window.

where ${X}_{i}$ is the signal value corresponding to sampling points. $P$ is the peak feature vector.

2.2.3.   Normalization

Min-max normalization was applied to the feature set by using Eq 9. And the features will be between 0 and 1.

where x represents the original eigenvalue and x' represents eigenvalue after normalization. The normalized EEG and fNIRS features are ${F}_{ij}^{EEG}$ and ${F}_{s}^{fNIRS}$.

2.2.4.   Hybrid feature fusion

The former feature fusion of hybrid only merges the EEG and fNIRS feature matrix. And the segmenting time tends to be consistent in hybrid: ${T}_{i}^{EEG} = {T}_{i}^{fNIRS}$. This method can decode the data classification performance at different time in the trial. However, the EEG signal collected is the electrical signal of the human brain, and the fNIRS signal is collected in the hemodynamic signal. There is a significant difference between the two devices in the way of physiological collection. In addition to exploring more dimensions of brain signals, there will also be some problems. Such as there is information delay between the two acquisition ways when the brain receives stimulation. It is means, the activation time of EEG and fNIRS signal does not match after brain activation. To explore that the segmenting time combination of EEG and fNIRS can affect the decoding task ability, we combine hybrid features in different segments.

where ${F}_{is}$ represents the fusion of CSP features in the ${i}^{th}$ time segment and statistical features in the ${s}^{th}$ time segment.

2.3.   Feature selection and classification

2.3.1.   Selection of crossing time windows

Mutual information (MI) is an index of interdependence between two random variables, which quantifies the amount of information contained in one random variable about the other. It can effectively express the nonlinear correlation between random variables, and it can be applied to rank features in feature selection ^[48]. The target of the MI method is to retain the features with the most discriminative and the less redundant information Feature F and label y are two random variables, The joint distribution functions of two random variables (F, y) are p(f, y); The marginal distribution functions are p(f) and p(y) respectively and the MI of random variables is $I(F, Y)$.

where F and Y have the more shared information when $I(F, Y)$ is larger, and vice versa.

Since EEG segments and fNIRS segments are recombined into crossing time windows, where F is extracted. Specifically, the crossing time windows can build $I\times S$ feature matrix. By calculating the MI value of feature matrix, the two-dimensional MI matrix $L\in {R}^{I\times S}$ is obtained. The crossing time windows corresponding to the largest MI represent the optimal crossing time window. The optimal crossing time window is selected for feature selection and classification in the next part.

2.3.2.   Feature selection and classification

Fisher criterion is a kind of statistic parameter that projects high-dimensional parameters into one dimension to measure class discriminant attributes. Fisher value represents the discrimination of samples. The discrimination is advanced with the increase of Fisher value ^[49]. The correlation coefficient is defined as follows:

where ${X}^{-}$ and${\mathrm{ }X}^{+}$ represents the samples of two class, $mean\left(\cdot \right)$is mean, $\mathit{var}\left(\cdot \right)$is variance.

Lasso was first proposed by Robert Tibshirani in 1996, which is a compression estimation method ^[50]. A more accurate model is obtained by constructing a penalty function to compress some regression coefficients. That means if the absolute values of the mandatory coefficients are less than the threshold and some regression coefficients will be set to zero ^[51]. The objective function of Lasso is as follows:

where f is feature vector, $F = [{{f}_{1}}^{T}, {{f}_{2}}^{T}, ..., {{f}_{n}}^{T}]$, y is label, $\beta$is regression coefficient of feature, $\stackrel{-}{\beta }$ is sparse regression coefficient of feature, $\lambda$ is penalty coefficient, the $\lambda$ larger is, the more regression coefficient will be set to zero in $\stackrel{-}{\beta }$. Finally, we save the feature that is not zero in $\stackrel{-}{\beta }$ and it is the sparse feature $\stackrel{-}{F}$.

The current sparse model and many L1 regularization models assume that the features are independent and ignore the inherent structural information in the feature set, such as linear and complex nonlinear relationships ^[52]. In addition, although some filter or wrapper methods can capture the non-linear relationship between features and response variables, they may not able to identify optimal features in the feature selection. In order to balance the linear and non-linear relationships between features and response variables, we propose a new feature selection framework named FLFS, which combined Fisher and Lasso into the framework to help identify the optimal feature set and improve the performance.

The steps of FLFS shown in Figure 2: Firstly, we calculate the Fisher value between each feature and label, denotes as ${I}_{m}$; Secondly, according to the absolute ${I}_{m}$, we rank all features. the features with higher coefficients have higher priority, and the first k features after sorting are extracted; Thirdly, the rest of features are sparse by Lasso, and the number of sparse feature sets is n. Finally, we fuse the first k feature sets selected based on MI and the feature sets after Lasso. Then the optimal feature set will be put in the classifier.

It is worth noting that $k, \lambda$ is the two hyperparameters in FLFS, where k means that select the first k features sorted $\lambda$ represents the penalty coefficient. When $k = 0$, FLFS is considered as Lasso; When $\lambda = 0$, FLFS is considered as Fisher. By setting $k, \lambda$, the optimal feature set can be selected, that is, the subset with the highest accuracy. The linear discriminant analysis (LDA) is adopted in this work since it has broad applications in classification. Plenty of hybrid BCI researches reported outstanding performance using LDA for classification. LDA also has the characteristics of simple principle and low computation. Deep learning is effective, but one of its limitations is that it requires numerous samples, so it may not work well on our dataset.

3.   Experimental

3.1.   Description of the datasets

In this paper, we use the public data set established by the Berlin Institute of Technology ^[53] to verify the proposed method. The distribution map of EEG electrodeposition and fNIRS photodiode position were shown in Figure 3. The dataset contains the mental arithmetic (MA) dataset.

The MA dataset included MA and baseline tasks and each subject performed 60 trials. Figure 4 shows the specific process of a trial, including two seconds of visual introduction, 10 seconds of task time, and 14–16 seconds of rest time. During the MA task, subjects were asked to subtract "one digit from" the "three digits" (for instance, 123-9) and repeatedly subtract "one digit" from the previous subtraction results until the task was completed. In MA tasks, baseline tasks were performed with rest.

3.2.   Data preprocessing

EEG data was filtered by a sixth-order zero-phase Butterworth filter and passbands were selected in the range of 4–35 Hz to eliminate interference and noise. In the fNIRS system, the single was filtered by a sixth-order zero-phase Butterworth filter with a passband of 0.01–0.1 Hz. Baseline correction was performed using the mean values of the concentration data of HBR and HBO 5 seconds before the trial. Setting of the time parameters for EEG and fNIRS:$L = 1, w = 3$. We employed a filter bank that bandpass filters the EEG into 4–18 Hz, 14–28 Hz and 24–35 Hz bands.

3.2.   Overall framework

In this part, we used $10\times 5$fold cross-validation method to increase the reliability of classification results. The Fisher value of each crossing time window was summed through the training set and the optimal crossing time window was selected for FLFS feature optimization. Parameters $k, \lambda$ were determined by cross-validation. The testing set used the determined $k, \lambda$ to select optimal feature subset and LDA classification. The overall framework is shown in Figure 5.

4.   Experimental results

For the above data set, before using the proposed method this paper, we use the Synchronization time windows (STW) to segment hybrid time data. The details of the segmentation are as follows: 0–3 s, 1–4 s, 2–5 s, 3–6 s, 4–7 s, 5–8 s, 6–9 s, 7–10 s. And the frequency band of EEG is divided into three sub-bands of 4–18 Hz, 14–28 Hz and 24–35 Hz. Table 1 shows the average accuracy of the MA task classification in the three modalities and different time segments for 10 subjects. The result shows that hybrid has a stronger classification accuracy in the three modes of EEG, fNIRS and hybrid. In addition, the classification accuracy of different periods is also significantly different. EEG, fNIRS and hybrid achieved the highest accuracy in 1–4 s (78.90 ± 1.47%), 7–10 s (76.44 ± 6.77%), and 7–10 s (83.33 ± 4.07%), respectively.

To demonstrate the advantages of the crossing time windows (CTW) to segment hybrid time data, we compare the STW and CTW for MA experiment. Table 2 shows the hybrid classification accuracy of 10 subjects under the CTW and STW. The result shows that the hybrid classification accuracy of 10 subjects using the CTW is better than the STW. And the average accuracy of STW and CTW is 87.72 ± 6.95% and 89.83 ± 6.48%.

Finally, to indicate the advantages of FLFS using the CTW, we employ three methods based on feature optimization: Fisher, LASSO, and FLFS. Table 3 shows the accuracy comparison of three different feature optimization methods (Fisher, LASSO, and FLFS) applied to 10 subjects on the MA dataset. The result shows that, compared with no feature optimization, using Fisher, LASSO, and FLFS feature optimization can all improve the accuracy of MA task classification. Specifically, the average accuracy rates of 10 subjects were 91.02 ± 5.97% (Fisher), 91.30 ± 6.27% (LASSO), and 92.52 ± 5.38% (FLFS).

5.   Discussion

As we all known, the hybrid acquisition method increases the dimensionality of the signal and improves the performance of the traditional single-mode method. However, there is no thought to consider the way of hybrid time combination in the task. Although the time for the subject to perform the task is determined, time associated between brain activity and task is undetermined. It means, we learn the time when the participant completes the task, but we do not know the time when the signal will active. Therefore, using a fixed single time for data interception and pattern recognition cannot achieve the optimal classification performance. It can be displayed from Table 1 that in different segmenting times, the classification accuracy of different modalities will fluctuate significantly. That also reflects the importance of the proposed CTW method in this paper. Table 2 shows the comparison between CTW and STW. The CTW combines multi-modal time windows plays an important role in proposed algorithm and improves the MA classification performance up to 2.11%.

In the experiment, this paper employs CTW and a sparse feature learning model to research the EEG and fNIRS data on the MA dataset. And we discuss that whether choosing the hybrid cross time windows can improve the accuracy of the classification task. The hybrid signals collected by 10 subjects are divided by the CTW. Following feature extraction and feature selection with FLFS, the LDA achieved the highest classification accuracy of 92.52 ± 5.38% in Table 3. The feature set which composed of the CSP feature and the statistical feature has a higher recognition ability. Figure 6 histogram displays that HBO features are mainly distributed in the Prefrontal area and HBR features are mainly distributed in the central area. This is consistent with the conclusion drawn by Shin et al. ^[53]. After feature selection, the optimized main features are distributed in the 4–28 Hz frequency band. In addition, only minority of EEG features are sparse, which displays that the features after EEG frequency division are of significance.

Considering the redundancy between signals and the computational time complexity, it is difficult to classify all crossing time windows. Therefore, we use mutual information to select the optimal crossing time window. Mutual information represents the discrimination of features, the greater the mutual information the greater the classification ability. Take subjects 2, 4, 5, and 6 as an instance, Figure 6 shows the calculation of mutual information correlation for the divided two-dimensional windows in the CTW experiment, Chroma is deepened with the increase of mutual information. We select the optimal crossing time windows with the largest mutual information value. Figure 7 shows that subject 2, 5, and 6 selects the first EEG time window (0–3 s), and the 8th fNIRS time window (7–10 s); subjects 4 select the first EEG to time window (5–8 s), and the 8th fNIRS time window (7–10 s). The phenomenon can be ranged into two principal situations. For majority of subjects, selecting the EEG time window precedes the selecting fNIRS time window and the time interval is large; but there are also a minority of subjects, although the selected EEG time window is also before the fNIRS time window, the time interval is near. The first situation appears it may be caused by the EEG and fNIRS achieve higher accuracy separately. The feature shows better classification performance, which means that the mutual information value corresponding to the EGG and fNIRS features are higher; The second situation appears it may be due to the inconsistency in the selection of hybrid time windows caused by individual differences.

In the extended feature set, the FLFS model can be used to collect the feature subset with the optimal classification accuracy. Taking subject 6 as an instance, a subset that provided the highest classification accuracy was identified by FLFS in the LDA model. This subset of features, shown in Figure 7, comprises peak and mean statistical features and CSP features. The pie chart shows that the initial proportions of EEG and fNIRS feature numbers were 92% and 8% of all feature set. However, under the feature selection, about 38% of the features number are sparse which can reduce the model calculation. After sparseness, the proportions of EEG and fNIRS features account for 55% and 6%, respectively. Compared with the proportion of features before feature selection, the feature proportion of the two modalities tend to be similar after feature selection.

6.   Conclusions

In this paper, a crossing time windows based FLFS optimization method for hybrid BCI systems is proposed. In our method, hybrid time segments were obtained from each time segmenting sample. After crossing and recombining each time window, the features were extracted using CSP and statistical method from each time window, which were combined to form a new feature vector. At last, feature selection algorithm and classification are applied to evaluate the effectiveness of the proposed method. Compared with the traditional synchronous selection time segment, the experimental results show that the method can increase accuracy rate of 4.8%. It also found that when the MI selects an optimal crossing time window, most subjects prefer the time windows that EEG and fNIRS are both activated at the same time. The proposed method has practical application prospects in HBCI. In the future, we will continue to undertake more investigations on this method using more datasets.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 61871427, 61971168 and U20B2074), National Key R & D Program of China for Intergovernmental International Science and Technology Innovation Cooperation Project (2017YFE0116800), and the Foundation of Zhejiang Provincial Education Department of China (No. Y202044279). The authors would like to acknowledge hybrid MA dataset collected by the Berlin Institute of Technology which were used to test the algorithms proposed in this study.

Conflict of interest

The authors declare there is no conflict of interest.