
Hybrid EEG-fNIRS brain-computer interface (HBCI) is widely employed to enhance BCI performance. EEG and fNIRS signals are combined to increase the dimensionality of the information. Time windows are used to select EEG and fNIRS singles synchronously. However, it ignores that specific modal signals have their own characteristics, when the task is stimulated, the information between the modalities will mismatch at the moment, which has a significant impact on the classification performance. Here we propose a novel crossing time windows optimization for mental arithmetic (MA) based BCI. The EEG and fNIRS signals were segmented separately by sliding time windows. Then crossing time windows (CTW) were combined with each one segment from EEG and fNIRS selected independently. Furthermore, EEG and fNIRS features were extracted using Filter Bank Common Spatial Pattern (FBCSP) and statistical methods from each sample. Mutual information was calculated for FBCSP and statistical features to characterize the discrimination of crossing time windows, and the optimal window would be selected based on the largest mutual information. Finally, a sparse structured framework of Fisher Lasso feature selection (FLFS) was designed to select the joint features, and conventional Linear Discriminant Analysis (LDA) was employed to perform classification. We used proposed method for a MA dataset. The classification accuracy of the proposed method is 92.52 ± 5.38% and higher than other methods, which shows the rationality and superiority of the proposed method.
Citation: Ming Meng, Luyang Dai, Qingshan She, Yuliang Ma, Wanzeng Kong. Crossing time windows optimization based on mutual information for hybrid BCI[J]. Mathematical Biosciences and Engineering, 2021, 18(6): 7919-7935. doi: 10.3934/mbe.2021392
[1] | Xu Yin, Ming Meng, Qingshan She, Yunyuan Gao, Zhizeng Luo . Optimal channel-based sparse time-frequency blocks common spatial pattern feature extraction method for motor imagery classification. Mathematical Biosciences and Engineering, 2021, 18(4): 4247-4263. doi: 10.3934/mbe.2021213 |
[2] | Xiaotong Ding, Lei Yang, Congsheng Li . Study of MI-BCI classification method based on the Riemannian transform of personalized EEG spatiotemporal features. Mathematical Biosciences and Engineering, 2023, 20(7): 12454-12471. doi: 10.3934/mbe.2023554 |
[3] | Xue Jia, Jing Xue, Shi-Yun Wang, Ji-Bo Wang . Polynomial time algorithm for minmax scheduling with common due-window and proportional-linear shortening processing times. Mathematical Biosciences and Engineering, 2022, 19(9): 8923-8934. doi: 10.3934/mbe.2022414 |
[4] | Weiguo Liu, Xuyin Wang, Xiaoxiao Wang, Peizhen Zhao . Due-window assignment scheduling with past-sequence-dependent setup times. Mathematical Biosciences and Engineering, 2022, 19(3): 3110-3126. doi: 10.3934/mbe.2022144 |
[5] | N Arunkumar, B Nagaraj, M Ruth Keziah . EpilepIndex: A novel feature engineering tool to detect epilepsy using EEG signals. Mathematical Biosciences and Engineering, 2023, 20(12): 21670-21691. doi: 10.3934/mbe.2023959 |
[6] | Haibo Li, Juncheng Tong . A novel clustering algorithm for time-series data based on precise correlation coefficient matching in the IoT. Mathematical Biosciences and Engineering, 2019, 16(6): 6654-6671. doi: 10.3934/mbe.2019331 |
[7] | Ying Xu, Jinyong Cheng . Secondary structure prediction of protein based on multi scale convolutional attention neural networks. Mathematical Biosciences and Engineering, 2021, 18(4): 3404-3422. doi: 10.3934/mbe.2021170 |
[8] | Tianhui Sha, Yikai Zhang, Yong Peng, Wanzeng Kong . Semi-supervised regression with adaptive graph learning for EEG-based emotion recognition. Mathematical Biosciences and Engineering, 2023, 20(6): 11379-11402. doi: 10.3934/mbe.2023505 |
[9] | Yanghan Ou, Siqin Sun, Haitao Gan, Ran Zhou, Zhi Yang . An improved self-supervised learning for EEG classification. Mathematical Biosciences and Engineering, 2022, 19(7): 6907-6922. doi: 10.3934/mbe.2022325 |
[10] | Jiacan Xu, Donglin Li, Peng Zhou, Chunsheng Li, Zinan Wang, Shenghao Tong . A multi-band centroid contrastive reconstruction fusion network for motor imagery electroencephalogram signal decoding. Mathematical Biosciences and Engineering, 2023, 20(12): 20624-20647. doi: 10.3934/mbe.2023912 |
Hybrid EEG-fNIRS brain-computer interface (HBCI) is widely employed to enhance BCI performance. EEG and fNIRS signals are combined to increase the dimensionality of the information. Time windows are used to select EEG and fNIRS singles synchronously. However, it ignores that specific modal signals have their own characteristics, when the task is stimulated, the information between the modalities will mismatch at the moment, which has a significant impact on the classification performance. Here we propose a novel crossing time windows optimization for mental arithmetic (MA) based BCI. The EEG and fNIRS signals were segmented separately by sliding time windows. Then crossing time windows (CTW) were combined with each one segment from EEG and fNIRS selected independently. Furthermore, EEG and fNIRS features were extracted using Filter Bank Common Spatial Pattern (FBCSP) and statistical methods from each sample. Mutual information was calculated for FBCSP and statistical features to characterize the discrimination of crossing time windows, and the optimal window would be selected based on the largest mutual information. Finally, a sparse structured framework of Fisher Lasso feature selection (FLFS) was designed to select the joint features, and conventional Linear Discriminant Analysis (LDA) was employed to perform classification. We used proposed method for a MA dataset. The classification accuracy of the proposed method is 92.52 ± 5.38% and higher than other methods, which shows the rationality and superiority of the proposed method.
Brain-computer interface (BCI) is a significant way of direct communication between the human central nervous system and the computer. It means, without the use of limbs, sounds, or any movements that require muscle activity, BCI can analyze brain induction signals which are related to the expected tasks. So BCI can generate commands to control external devices [1,2,3]. BCI systems also consist of signal acquisition, pre-processing, feature extraction, classification, external communication with the device, and feedback stages.
Electroencephalogram (EEG) [4,5], Electrocardiogram (ECoG) [6], Functional near-infrared spectroscopy(fNIRS) [7,8,9], Functional magnetic resonance imaging(fMRI) [10,11] and Magnetoencephalography (MEG) [12] are common signal acquisition modality in BCI systems. EEG is a bioelectrical imaging method that uses scalp electrodes to measure the voltage fluctuation caused by the electrical activity of cortical neurons. EEG stands out because of its high time resolution, frequency-domain characteristics, convenience, and safety [13,14]. Due to volume conduction effect in the cerebral cortex, it is difficult to determine whether EEG signal is generated near the surface of the cerebral cortex or in deeper areas. The spatial resolution of EEG is relatively low and it is susceptible to electrical noise and motion artifacts [15,16]. Hence, the classification accuracy of EEG is seriously damaged. fNIRS is an optical imaging modality that evaluates the hemodynamic activity in the brain. fNIRS can be used to measure the concentration changes of deoxyhemoglobin (HBR) and oxyhemoglobin (HBO) [17,18,19,20,21]. fNIRS stands out because of its non-invasive, portability, safety, and relatively high spatial resolution [22,23]. However, due to the slow response of blood oxygen concentration, the temporal resolution of fNIRS is low.
In the past few years, researchers have proposed hybrid BCI systems to overcome the limitations of single-mode systems [24,25,26,27]. It is feasible and reliable to use EEG and fNIRS as hybrid BCI to detect physiological signals of the specific subject. And hybrid BCI is widely applied in many cognitive rehabilitation fields [28,29,30]. Since EEG and fNIRS are specific time domain information rich, many researchers analyze the time domain of EEG and fNIRS [31,32]. Previous approaches were to segment synchronous EEG and fNIRS data to analyze the physiological state of the brain in different periods of the task [33,34]. And different time windows of signal have a significant impact on classification performance [35]. However, it ignores that the specific modal signals have various physiological characteristics. There are different matching degrees of time-domain information between EEG and fNIRS in the acquisition process. EEG signals reflect electrical activity originating as a result of neuronal firing when each task or activity is performed [36]. On the other hand, hemodynamic activity appears in the form of blood flow changes that result from neuronal firing [37,38]. Therefore, the electrical signal represented by EEG and the blood oxygen signal represented by fNIRS will not be activated at the same time when the subjects are stimulated by the task.
To solve the above problems, a crossing time windows optimal method based on Fisher Lasso feature selection (FLFS) algorithm for hybrid BCI is proposed. Firstly, the proposed method consists of multi-time segmenting and temporal band-pass filtering. Secondly, EEG and fNIRS time windows are crossed and combined. Thirdly, FBCSP and statistical methods are used to extract EEG and fNIRS features respectively. To select optimal time window, all features in each time window are sorted based on mutual information (MI). Finally, the features are put into FLFS and LDA for feature selection and classification.
Parts of this paper are arranged as follows: The second section describes the preprocessing method; the third section describes the dataset and experimental results; the fourth section discusses the experimental results; and the last part summarizes this full paper.
As shown in Figure 1, it is a crossing time optimal combination method based on sparse feature selection for hybrid BCI. This method mainly includes the multi-time segmenting of EEG and fNIRS data; then FBCSP and statistical method are the hybrid feature extraction method; Finally, feature selection and classification are executed.
In the first part, EEG raw signals are decomposed to a total of I time segments. The time segments are Ti=[L×i,L×i+w],0≤i≤I, where the unit is second and L denotes step size of the window, w denotes length of the window. In the fNIRS system, the fluctuating concentration of HBO and HBR are converted from the measured light intensity by the modified Beer-Lambert law [39,40]. And they are not obvious in frequency but have good time characteristics. Therefore, this part only segments the time of the fNIRS signal, and the time segmenting is the same as EEG. Fs∈Rc×k means the fNIRS signal in the sth time segment, where k is the sampling point and k=1,2,...,K, c represents the number of channels.
CSP is a spatial filtering algorithm for two classes of tasks, which can extract the spatial distribution components of each class from multi-channel EEG data. X1 and X2 are signal matrices for two classes of tasks respectively [41]. The covariance of the signal matrix is calculated and normalized:
R1=X1XT1trace(X1,XT1)R2=X2XT2trace(X2,XT2) | (1) |
where trace(⋅) is the trace of the matrix. Then find the covariance matrix R of the mixed space and decompose eigenvalue:
R=−R1+−R2=UΛUT | (2) |
where −Ri is the covariance matrix of two classes of task and i=1,2. U is eigenvectors of matrix R, Λ is eigenvalue matrix. So the whitening matrix P can be obtained:
P=√Λ−1UT | (3) |
Then matrix R1 and R2 whiten and used by principal component analysis(PCA):
S1=PR1PT=B1Λ1BT1S2=PR2PT=B2Λ2BT2 | (4) |
where B1,B2 are the eigenvectors of a matrix S1, S2 and B1=B2. Λ1+Λ2=I, where I is the identity matrix. The sum of the eigenvalues is 1. When the eigenvalue of one class is the largest and the other is the smallest. It is the reason that the optimal spatial filter W can distinguish two-class samples.
W=BTP | (5) |
where W∈Rc×2m. Finally, the feature vector f with high ability of discrimination is obtained.
fij=log(var(WTEij)) | (6) |
where fij∈R1×2m represents the feature vector extracted by CSP from the EEG signal in the jth frequency band and the ith time segment.
Since the performance of the spatial filter in the CSP algorithm depends on the frequency band of the EEG, FBCSP is to first pass the EEG signal through a bandpass filter to form multiple bands and then extract the CSP features from these sub-bands [42]. Therefore, the segmenting data is put into the third-order Butterworth band-pass filter to divide the multi-band.
The statistical features of fNIRS were employed by signal mean [43], signal slope [44], signal variance [45], signal peak [45,46], signal kurtosis and signal skewness. Signal mean and the signal peak will be applied in this part [47].
Signal mean
The average concentration of HBO and HBR is calculated according to the Eq 7:
Ms=1N∑Ni=1Xi | (7) |
where N is the number of sampling point and X is the HBO or HBR data. M is the mean feature vector and s denotes the fNIRS data samples in the sth time segment.
Signal peak
The signal peak is the maximum value of the signal in the time window.
Ps=maxXi,1≤i≤N | (8) |
where Xi is the signal value corresponding to sampling points. P is the peak feature vector.
Min-max normalization was applied to the feature set by using Eq 9. And the features will be between 0 and 1.
x'=x−min(x)max(x)−min(x) | (9) |
where x represents the original eigenvalue and x' represents eigenvalue after normalization. The normalized EEG and fNIRS features are FEEGij and FfNIRSs.
The former feature fusion of hybrid only merges the EEG and fNIRS feature matrix. And the segmenting time tends to be consistent in hybrid: TEEGi=TfNIRSi. This method can decode the data classification performance at different time in the trial. However, the EEG signal collected is the electrical signal of the human brain, and the fNIRS signal is collected in the hemodynamic signal. There is a significant difference between the two devices in the way of physiological collection. In addition to exploring more dimensions of brain signals, there will also be some problems. Such as there is information delay between the two acquisition ways when the brain receives stimulation. It is means, the activation time of EEG and fNIRS signal does not match after brain activation. To explore that the segmenting time combination of EEG and fNIRS can affect the decoding task ability, we combine hybrid features in different segments.
Fis=[FEEGi1,FEEGi2,⋅⋅⋅,FEEGiJ,FfNIRSs],0≤i≤I,0≤s≤S | (10) |
where Fis represents the fusion of CSP features in the ith time segment and statistical features in the sth time segment.
Mutual information (MI) is an index of interdependence between two random variables, which quantifies the amount of information contained in one random variable about the other. It can effectively express the nonlinear correlation between random variables, and it can be applied to rank features in feature selection [48]. The target of the MI method is to retain the features with the most discriminative and the less redundant information Feature F and label y are two random variables, The joint distribution functions of two random variables (F, y) are p(f, y); The marginal distribution functions are p(f) and p(y) respectively and the MI of random variables is I(F,Y).
I(F,Y)=∑y∈Y∑f∈Fp(f,y)log(p(f,y)p(f)p(y)) | (11) |
where F and Y have the more shared information when I(F,Y) is larger, and vice versa.
Since EEG segments and fNIRS segments are recombined into crossing time windows, where F is extracted. Specifically, the crossing time windows can build I×S feature matrix. By calculating the MI value of feature matrix, the two-dimensional MI matrix L∈RI×S is obtained. The crossing time windows corresponding to the largest MI represent the optimal crossing time window. The optimal crossing time window is selected for feature selection and classification in the next part.
Fisher criterion is a kind of statistic parameter that projects high-dimensional parameters into one dimension to measure class discriminant attributes. Fisher value represents the discrimination of samples. The discrimination is advanced with the increase of Fisher value [49]. The correlation coefficient is defined as follows:
scoreF(X)=(mean(X−)−mean(X+))2var(X−)+var(X+) | (12) |
where X− andX+ represents the samples of two class, mean(⋅)is mean, var(⋅)is variance.
Lasso was first proposed by Robert Tibshirani in 1996, which is a compression estimation method [50]. A more accurate model is obtained by constructing a penalty function to compress some regression coefficients. That means if the absolute values of the mandatory coefficients are less than the threshold and some regression coefficients will be set to zero [51]. The objective function of Lasso is as follows:
−β=argminβ(‖y−Fβ‖22+λ‖β‖1) | (13) |
where f is feature vector, F=[f1T,f2T,...,fnT], y is label, βis regression coefficient of feature, −β is sparse regression coefficient of feature, λ is penalty coefficient, the λ larger is, the more regression coefficient will be set to zero in −β. Finally, we save the feature that is not zero in −β and it is the sparse feature −F.
The current sparse model and many L1 regularization models assume that the features are independent and ignore the inherent structural information in the feature set, such as linear and complex nonlinear relationships [52]. In addition, although some filter or wrapper methods can capture the non-linear relationship between features and response variables, they may not able to identify optimal features in the feature selection. In order to balance the linear and non-linear relationships between features and response variables, we propose a new feature selection framework named FLFS, which combined Fisher and Lasso into the framework to help identify the optimal feature set and improve the performance.
The steps of FLFS shown in Figure 2: Firstly, we calculate the Fisher value between each feature and label, denotes as Im; Secondly, according to the absolute Im, we rank all features. the features with higher coefficients have higher priority, and the first k features after sorting are extracted; Thirdly, the rest of features are sparse by Lasso, and the number of sparse feature sets is n. Finally, we fuse the first k feature sets selected based on MI and the feature sets after Lasso. Then the optimal feature set will be put in the classifier.
It is worth noting that k,λ is the two hyperparameters in FLFS, where k means that select the first k features sorted λ represents the penalty coefficient. When k=0, FLFS is considered as Lasso; When λ=0, FLFS is considered as Fisher. By setting k,λ, the optimal feature set can be selected, that is, the subset with the highest accuracy. The linear discriminant analysis (LDA) is adopted in this work since it has broad applications in classification. Plenty of hybrid BCI researches reported outstanding performance using LDA for classification. LDA also has the characteristics of simple principle and low computation. Deep learning is effective, but one of its limitations is that it requires numerous samples, so it may not work well on our dataset.
In this paper, we use the public data set established by the Berlin Institute of Technology [53] to verify the proposed method. The distribution map of EEG electrodeposition and fNIRS photodiode position were shown in Figure 3. The dataset contains the mental arithmetic (MA) dataset.
The MA dataset included MA and baseline tasks and each subject performed 60 trials. Figure 4 shows the specific process of a trial, including two seconds of visual introduction, 10 seconds of task time, and 14–16 seconds of rest time. During the MA task, subjects were asked to subtract "one digit from" the "three digits" (for instance, 123-9) and repeatedly subtract "one digit" from the previous subtraction results until the task was completed. In MA tasks, baseline tasks were performed with rest.
EEG data was filtered by a sixth-order zero-phase Butterworth filter and passbands were selected in the range of 4–35 Hz to eliminate interference and noise. In the fNIRS system, the single was filtered by a sixth-order zero-phase Butterworth filter with a passband of 0.01–0.1 Hz. Baseline correction was performed using the mean values of the concentration data of HBR and HBO 5 seconds before the trial. Setting of the time parameters for EEG and fNIRS:L=1,w=3. We employed a filter bank that bandpass filters the EEG into 4–18 Hz, 14–28 Hz and 24–35 Hz bands.
In this part, we used 10×5fold cross-validation method to increase the reliability of classification results. The Fisher value of each crossing time window was summed through the training set and the optimal crossing time window was selected for FLFS feature optimization. Parameters k,λ were determined by cross-validation. The testing set used the determined k,λ to select optimal feature subset and LDA classification. The overall framework is shown in Figure 5.
For the above data set, before using the proposed method this paper, we use the Synchronization time windows (STW) to segment hybrid time data. The details of the segmentation are as follows: 0–3 s, 1–4 s, 2–5 s, 3–6 s, 4–7 s, 5–8 s, 6–9 s, 7–10 s. And the frequency band of EEG is divided into three sub-bands of 4–18 Hz, 14–28 Hz and 24–35 Hz. Table 1 shows the average accuracy of the MA task classification in the three modalities and different time segments for 10 subjects. The result shows that hybrid has a stronger classification accuracy in the three modes of EEG, fNIRS and hybrid. In addition, the classification accuracy of different periods is also significantly different. EEG, fNIRS and hybrid achieved the highest accuracy in 1–4 s (78.90 ± 1.47%), 7–10 s (76.44 ± 6.77%), and 7–10 s (83.33 ± 4.07%), respectively.
TIME(s) | Modality Type | |
EEG | fNIRS | |
0–3 | 78.9 | 67.2 |
1–4 | 80.3 | 69.1 |
2–5 | 78.1 | 72.0 |
3–6 | 79.2 | 75.0 |
4–7 | 79.4 | 78.0 |
5–8 | 80.2 | 80.9 |
6–9 | 79.4 | 83.4 |
7–10 | 75.7 | 85.9 |
Mean ± std | 78.90 ± 1.47 | 76.44 ± 6.77 |
To demonstrate the advantages of the crossing time windows (CTW) to segment hybrid time data, we compare the STW and CTW for MA experiment. Table 2 shows the hybrid classification accuracy of 10 subjects under the CTW and STW. The result shows that the hybrid classification accuracy of 10 subjects using the CTW is better than the STW. And the average accuracy of STW and CTW is 87.72 ± 6.95% and 89.83 ± 6.48%.
Subject | Methods | |
STW | CTW | |
S1 | 88.2 | 92.5 |
S2 | 88.0 | 86.3 |
S3 | 92.5 | 91.8 |
S4 | 76.2 | 79.7 |
S5 | 78.0 | 80.2 |
S6 | 91.5 | 92.7 |
S7 | 85.2 | 92.5 |
S8 | 93.3 | 96.3 |
S9 | 85.3 | 86.8 |
S10 | 99.0 | 99.5 |
Mean ± std | 87.72±6.95 | 89.83±6.48 |
Finally, to indicate the advantages of FLFS using the CTW, we employ three methods based on feature optimization: Fisher, LASSO, and FLFS. Table 3 shows the accuracy comparison of three different feature optimization methods (Fisher, LASSO, and FLFS) applied to 10 subjects on the MA dataset. The result shows that, compared with no feature optimization, using Fisher, LASSO, and FLFS feature optimization can all improve the accuracy of MA task classification. Specifically, the average accuracy rates of 10 subjects were 91.02 ± 5.97% (Fisher), 91.30 ± 6.27% (LASSO), and 92.52 ± 5.38% (FLFS).
Subject | Methods | ||
Fisher | LASSO | FLFS | |
S1 | 93.2 | 92.6 | 93.3 |
S2 | 89.6 | 87.7 | 90.2 |
S3 | 92.3 | 95.5 | 95.7 |
S4 | 84.3 | 83.2 | 85.8 |
S5 | 81.0 | 80.3 | 83.3 |
S6 | 93.3 | 95.7 | 95.9 |
S7 | 92.5 | 92.3 | 93.2 |
S8 | 98.2 | 97.0 | 98.5 |
S9 | 85.8 | 88.7 | 89.3 |
S10 | 100.0 | 100.0 | 100.0 |
Mean ± std | 91.02 ± 5.97 | 91.30 ± 6.27 | 92.52 ± 5.38 |
As we all known, the hybrid acquisition method increases the dimensionality of the signal and improves the performance of the traditional single-mode method. However, there is no thought to consider the way of hybrid time combination in the task. Although the time for the subject to perform the task is determined, time associated between brain activity and task is undetermined. It means, we learn the time when the participant completes the task, but we do not know the time when the signal will active. Therefore, using a fixed single time for data interception and pattern recognition cannot achieve the optimal classification performance. It can be displayed from Table 1 that in different segmenting times, the classification accuracy of different modalities will fluctuate significantly. That also reflects the importance of the proposed CTW method in this paper. Table 2 shows the comparison between CTW and STW. The CTW combines multi-modal time windows plays an important role in proposed algorithm and improves the MA classification performance up to 2.11%.
In the experiment, this paper employs CTW and a sparse feature learning model to research the EEG and fNIRS data on the MA dataset. And we discuss that whether choosing the hybrid cross time windows can improve the accuracy of the classification task. The hybrid signals collected by 10 subjects are divided by the CTW. Following feature extraction and feature selection with FLFS, the LDA achieved the highest classification accuracy of 92.52 ± 5.38% in Table 3. The feature set which composed of the CSP feature and the statistical feature has a higher recognition ability. Figure 6 histogram displays that HBO features are mainly distributed in the Prefrontal area and HBR features are mainly distributed in the central area. This is consistent with the conclusion drawn by Shin et al. [53]. After feature selection, the optimized main features are distributed in the 4–28 Hz frequency band. In addition, only minority of EEG features are sparse, which displays that the features after EEG frequency division are of significance.
Considering the redundancy between signals and the computational time complexity, it is difficult to classify all crossing time windows. Therefore, we use mutual information to select the optimal crossing time window. Mutual information represents the discrimination of features, the greater the mutual information the greater the classification ability. Take subjects 2, 4, 5, and 6 as an instance, Figure 6 shows the calculation of mutual information correlation for the divided two-dimensional windows in the CTW experiment, Chroma is deepened with the increase of mutual information. We select the optimal crossing time windows with the largest mutual information value. Figure 7 shows that subject 2, 5, and 6 selects the first EEG time window (0–3 s), and the 8th fNIRS time window (7–10 s); subjects 4 select the first EEG to time window (5–8 s), and the 8th fNIRS time window (7–10 s). The phenomenon can be ranged into two principal situations. For majority of subjects, selecting the EEG time window precedes the selecting fNIRS time window and the time interval is large; but there are also a minority of subjects, although the selected EEG time window is also before the fNIRS time window, the time interval is near. The first situation appears it may be caused by the EEG and fNIRS achieve higher accuracy separately. The feature shows better classification performance, which means that the mutual information value corresponding to the EGG and fNIRS features are higher; The second situation appears it may be due to the inconsistency in the selection of hybrid time windows caused by individual differences.
In the extended feature set, the FLFS model can be used to collect the feature subset with the optimal classification accuracy. Taking subject 6 as an instance, a subset that provided the highest classification accuracy was identified by FLFS in the LDA model. This subset of features, shown in Figure 7, comprises peak and mean statistical features and CSP features. The pie chart shows that the initial proportions of EEG and fNIRS feature numbers were 92% and 8% of all feature set. However, under the feature selection, about 38% of the features number are sparse which can reduce the model calculation. After sparseness, the proportions of EEG and fNIRS features account for 55% and 6%, respectively. Compared with the proportion of features before feature selection, the feature proportion of the two modalities tend to be similar after feature selection.
In this paper, a crossing time windows based FLFS optimization method for hybrid BCI systems is proposed. In our method, hybrid time segments were obtained from each time segmenting sample. After crossing and recombining each time window, the features were extracted using CSP and statistical method from each time window, which were combined to form a new feature vector. At last, feature selection algorithm and classification are applied to evaluate the effectiveness of the proposed method. Compared with the traditional synchronous selection time segment, the experimental results show that the method can increase accuracy rate of 4.8%. It also found that when the MI selects an optimal crossing time window, most subjects prefer the time windows that EEG and fNIRS are both activated at the same time. The proposed method has practical application prospects in HBCI. In the future, we will continue to undertake more investigations on this method using more datasets.
This work was supported by the National Natural Science Foundation of China (No. 61871427, 61971168 and U20B2074), National Key R & D Program of China for Intergovernmental International Science and Technology Innovation Cooperation Project (2017YFE0116800), and the Foundation of Zhejiang Provincial Education Department of China (No. Y202044279). The authors would like to acknowledge hybrid MA dataset collected by the Berlin Institute of Technology which were used to test the algorithms proposed in this study.
The authors declare there is no conflict of interest.
[1] |
J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, T. M. Vaughan, Brain-computer interfaces for communication and control, Clin. Neurophysiol., 113 (2002), 767-791. doi: 10.1016/S1388-2457(02)00057-3
![]() |
[2] |
L. F. Nicolas-Alonso, J. Gomez-Gil, Brain Computer Interfaces, a Review, Sensors, 12 (2012), 1211-1279. doi: 10.3390/s120201211
![]() |
[3] |
A. D. Bigirimana, N. Siddique, D. Coyle, Emotion-Inducing Imagery Versus Motor Imagery for a Brain-Computer Interface, IEEE Trans. Neural Syst. Rehab. Eng., 28 (2020), 850-859. doi: 10.1109/TNSRE.2020.2978951
![]() |
[4] |
H. Azizollahi, M. Darbas, M. M. Diallo, A. El Badia, S. Lohrengel, Eeg in Neonates: Forward Modeling and Sensitivity Analysis with Respect to Variations of the Conductivity, Math. Biosci. Eng., 15 (2018), 905-932. doi: 10.3934/mbe.2018041
![]() |
[5] |
J. W. Choi, S. Huh, S. Jo, Improving performance in motor imagery BCI-based control applications via virtually embodied feedback, Comput. Biol. Med., 127 (2020), 104079-104088. doi: 10.1016/j.compbiomed.2020.104079
![]() |
[6] | J. Unterweger, M. Seeber, S. Zanos, J. G. Ojemann, R. Scherer, ECoG Beta Suppression and Modulation During Finger Extension and Flexion, Front. Neurosci., 14 (2020), 35-45. |
[7] |
R. Rosas-Romero, E. Guevara, K. Peng, D. K. Nguyen, F. Lesage, P. Pouliot, W. E. Lima-Saad, Prediction of epileptic seizures with convolutional neural networks and functional near-infrared spectroscopy signals, Comput. Biol. Med., 111 (2019), 103355-103365. doi: 10.1016/j.compbiomed.2019.103355
![]() |
[8] |
E. A. Aydin, Subject-Specific feature selection for near infrared spectroscopy based brain-computer interfaces, Comput. Methods Programs Biomed., 195 (2020), 105535-105547. doi: 10.1016/j.cmpb.2020.105535
![]() |
[9] |
C. G. Li, M. Su, J. C. Xu, H. D. Jin, L. N. Sun, A Between-Subject fNIRS-BCI Study on Detecting Self-Regulated Intention During Walking, IEEE Trans. Neural Syst. Rehab. Eng., 28 (2020), 531-540. doi: 10.1109/TNSRE.2020.2965628
![]() |
[10] | G. Lioi, S. Butet, M. Fleury, E. Bannier, A. Lecuyer, I. Bonan, C. Barillot, A Multi-Target Motor Imagery Training Using Bimodal EEG-fMRI Neurofeedback: A Pilot Study in Chronic Stroke Patients, Front. Hum. Neurosci., 14 (2020), 37-50. |
[11] |
G. Valente, A. L. Kaas, E. Formisano, R. Goebela, Optimizing fMRI experimental design for MVPA-based BCI control: Combining the strengths of block and event-related designs, NeuroImage, 186 (2019), 369-381. doi: 10.1016/j.neuroimage.2018.10.080
![]() |
[12] |
J. Mellinger, G. Schalk, C. Braun, H. Preissl, W. Rosenstiel, N. Birbaumer, A. Kubler, An MEG-based brain-computer interface (BCI), NeuroImage, 36 (2007), 581-593. doi: 10.1016/j.neuroimage.2007.03.019
![]() |
[13] |
Q. Noirhomme, R. I. Kitney, B. Macq, Single-trial EEG source reconstruction for brain-computer interface, IEEE Trans. Biomed. Eng., 55 (2008), 1592-1601. doi: 10.1109/TBME.2007.913986
![]() |
[14] |
M. Arvaneh, C. T. Guan, K. K. Ang, C. Quek, Optimizing the Channel Selection and Classification Accuracy in EEG-Based BCI, IEEE Trans. Biomed. Eng., 58 (2011), 1865-1873. doi: 10.1109/TBME.2011.2131142
![]() |
[15] | S. R. Soekadar, M. Witkowski, E. G. Cossio, N. Birbaumer, L. G. Cohen, Learned EEG-based brain self-regulation of motor-related oscillations during application of transcranial electric brain stimulation: feasibility and limitations, Front. Behav. Neurosci., 8 (2014), 93-102. |
[16] | D. Wang, D. Q. Miao, G. Blohm, Multi-class motor imagery EEG decoding for brain-computer interfaces, Front. Neurosci., 6 (2012), 151-164. |
[17] |
A. Villringer, J. Planck, C. Hock, L. Schleinkofer, U. Dirnagl, Near infrared spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults, Neurosci. Lett., 154 (1993), 101-104. doi: 10.1016/0304-3940(93)90181-J
![]() |
[18] |
Y. Hoshi, H. Onoe, Y. Watanabe, J. Andersson, M. Bergstrom, A. Lilja, B. Langstrom, M. Tamura, Non-synchronous behavior of neuronal activity, oxidative metabolism and blood supply during mental tasks in man, Neurosci. Lett., 172 (1994), 129-133. doi: 10.1016/0304-3940(94)90679-3
![]() |
[19] |
Y. Hoshi, M. Tamura, Near-infrared optical detection of sequential brain activation in the prefrontal cortex during mental tasks, NeuroImage, 5 (1997), 292-297. doi: 10.1006/nimg.1997.0270
![]() |
[20] |
A. Villringer, B. Chance, Non-invasive optical spectroscopy and imaging of human brain function, Trends Neurosci., 20 (1997), 435-442. doi: 10.1016/S0166-2236(97)01132-6
![]() |
[21] |
K. S. Hong, H. D. Nguyen, State-space models of impulse hemodynamic responses over motor, somatosensory, and visual cortices, Biomed. Opt. Express, 5 (2014), 1778-1798. doi: 10.1364/BOE.5.001778
![]() |
[22] |
L. F. Nicolas-Alonso, J. Gomez-Gil, Brain computer interfaces, a review, Sensors (Basel), 12 (2012), 1211-1279. doi: 10.3390/s120201211
![]() |
[23] | N. Naseer, N. K. Qureshi, F. M. Noori, K. S. Hong, Analysis of Different Classification Techniques for Two-Class Functional Near-Infrared Spectroscopy-Based Brain-Computer Interface, Comput. Intell. Neurosci., 2016 (2016), 5480760-5480771. |
[24] |
S. Fazli, J. Mehnert, J. Steinbrink, G. Curio, A. Villringer, K. R. Muller, B. Blankertz, Enhanced performance by a hybrid NIRS-EEG brain computer interface, NeuroImage, 59 (2012), 519-529. doi: 10.1016/j.neuroimage.2011.07.084
![]() |
[25] |
S. Firooz, S. K. Setarehdan, IQ estimation by means of EEG-fNIRS recordings during a logical mathematical intelligence test, Comput. Biol. Med., 110 (2019), 218-226. doi: 10.1016/j.compbiomed.2019.05.017
![]() |
[26] |
Y. Tomita, F. B. Vialatte, G. Dreyfus, Y. Mitsukura, H. Bakardjian, A. Cichocki, Bimodal BCI Using Simultaneously NIRS and EEG, IEEE Trans. Biomed. Eng., 61 (2014), 1274-1284. doi: 10.1109/TBME.2014.2300492
![]() |
[27] | A. P. Buccino, H. O. Keles, A. Omurtag, Hybrid EEG-fNIRS Asynchronous Brain-Computer Interface for Multiple Motor Tasks, Plos One, 11 (2016), 0146610-0146627. |
[28] | A. Berger, F. Horst, S. Müller, F. Steinberg, M. Doppelmayr, Current State and Future Prospects of EEG and fNIRS in Robot-Assisted Gait Rehabilitation: A Brief Review, Front. Hum. Neurosci., 13 (2019), 172-172. |
[29] |
S. K. Yeom, D.-O. Won, S. I. Chi, K.-S. Seo, H. J. Kim, K. R. Müller, S. W. Lee, Spatio-temporal dynamics of multimodal EEG-fNIRS signals in the loss and recovery of consciousness under sedation using midazolam and propofol, PLoS One, 12 (2017), e0187743. doi: 10.1371/journal.pone.0177197
![]() |
[30] | S. Mandal, B. K. Singh, K. Thakur, Classification of working memory loads using hybrid EEG and fNIRS in machine learning paradigm, ElL, 56 (2020), 1386-1388. |
[31] |
Y. Zhang, C. S. Nam, G. X. Zhou, J. Jin, X. Y. Wang, A. Cichocki, Temporally Constrained Sparse Group Spatial Patterns for Motor Imagery BCI, IEEE Trans. Cyber., 49 (2019), 3322-3332. doi: 10.1109/TCYB.2018.2841847
![]() |
[32] | S. B. Borgheai, M. Abtahi, K. Mankodiya, J. McLinden, Y. Shahriari, Towards a Single Trial fNIRS-based Brain-Computer Interface for Communication, IEEE Int. Joint Conf. Neural Eng., 2019 (2019), 1030-1033. |
[33] | M. J. Khan, K. S. Hong, Hybrid EEG-fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control, Front. Neurorobot., 11 (2017), 6-19. |
[34] |
S. Fazli, J. Mehnert, J. Steinbrink, G. Curio, A. Villringer, K. R. Muller, B. Blankertz, Enhanced performance by a hybrid NIRS-EEG brain computer interface, NeuroImage, 59 (2012), 519-529. doi: 10.1016/j.neuroimage.2011.07.084
![]() |
[35] |
F. M. Noori, N. Naseer, N. K. Qureshi, H. Nazeer, R. A. Khan, Optimal feature selection from fNIRS signals using genetic algorithms for BCI, Neurosci. Lett., 647 (2017), 61-66. doi: 10.1016/j.neulet.2017.03.013
![]() |
[36] |
P. Olejniczak, Neurophysiologic basis of EEG, Clin. Neurophysiol., 23 (2006), 186-189. doi: 10.1097/01.wnp.0000220079.61973.6c
![]() |
[37] | F. Matthews, B. A. Pearlmutter, T. E. Wards, C. Soraghan, C. Markham, Hemodynamics for Brain-Computer Interfaces, ISPM, 25 (2008), 87-94. |
[38] |
B. K. Min, M. J. Marzelli, S. S. Yoo, Neuroimaging-based approaches in the brain-computer interface, Trends Biotechnol., 28 (2010), 552-560. doi: 10.1016/j.tibtech.2010.08.002
![]() |
[39] |
M. Essenpreis, C. E. Elwell, M. Cope, P. van der Zee, S. R. Arridge, D. T. Delpy, Spectral dependence of temporal point spread functions in human tissues, Appl. Opt., 32 (1993), 418-425. doi: 10.1364/AO.32.000418
![]() |
[40] |
F. Scholkmann, S. Kleiser, A. J. Metz, R. Zimmermann, J. M. Pavia, U. Wolf, M. Wolf, A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology, Neuroimage, 85 (2014), 6-27. doi: 10.1016/j.neuroimage.2013.05.004
![]() |
[41] |
Y. Wang, P. Berg, M. Scherg, Common spatial subspace decomposition applied to analysis of brain responses under multiple task conditions: a simulation study, Clin. Neurophysiol., 110 (1999), 604-614. doi: 10.1016/S1388-2457(98)00056-X
![]() |
[42] | K. K. Ang, Z. Y. Chin, H. H. Zhang, C. T. Guan, Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface, IEEE Int. Joint Conf. Neural Networks, 2008 (2008), 2390-2397. |
[43] |
R. Sitaram, H. H. Zhang, C. T. Guan, M. Thulasidas, Y. Hoshi, A. Ishikawa, et al., Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface, Neuroimage, 34 (2007), 1416-1427. doi: 10.1016/j.neuroimage.2006.11.005
![]() |
[44] |
K. S. Hong, N. Naseer, Y. H. Kim, Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI, Neurosci. Lett., 587 (2015), 87-92. doi: 10.1016/j.neulet.2014.12.029
![]() |
[45] |
L. Holper, M. Wolf, Single-trial classification of motor imagery differing in task complexity: a functional near-infrared spectroscopy study, J. Neuroeng. Rehabilitation, 8 (2011), 34-47. doi: 10.1186/1743-0003-8-34
![]() |
[46] |
G. Bauernfeind, R. Scherer, G. Pfurtscheller, C. Neuper, Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic, Med. Biol. Eng. Comput., 49 (2011), 979-984. doi: 10.1007/s11517-011-0792-5
![]() |
[47] | N. Naseer, F. M. Noori, N. K. Qureshi, K. S. Hong, Determining Optimal Feature-Combination for LDA Classification of Functional Near-Infrared Spectroscopy Signals in Brain-Computer Interface Application, Front. Hum. Neurosci., 10 (2016), 237-247. |
[48] | H. C. Peng, F. H. Long, C. Ding, Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy, ITPAM, 27 (2005), 1226-1238. |
[49] | K. R. Müller, M. Krauledat, G. Dornhege, G. Curio, B. J. B. T. Blankertz, Machine learning techniques for brain-computer interfaces, IEEE. Trans. Biomed. Eng., 49 (2004), 11-22. |
[50] |
R. Tibshirani, Regression shrinkage and selection via the lasso: a retrospective, J. R. Stat. Soc. Series B Stat. Methodol., 73 (2011), 273-282. doi: 10.1111/j.1467-9868.2011.00771.x
![]() |
[51] |
L. Meier, S. A. van de Geer, P. Buhlmann, The group lasso for logistic regression, J. R. Stat. Soc. Series B Stat. Methodol., 70 (2008), 53-71. doi: 10.1111/j.1467-9868.2007.00627.x
![]() |
[52] |
R. Hosseini, B. Walsh, F. H. Tian, S. Y. Wang, An fNIRS-Based Feature Learning and Classification Framework to Distinguish Hemodynamic Patterns in Children Who Stutter, IEEE Trans. Neural Syst. Rehab. Eng., 26 (2018), 1254-1263. doi: 10.1109/TNSRE.2018.2829083
![]() |
[53] |
J. Shin, A. von Luhmann, B. Blankertz, D. W. Kim, J. Jeong, H. J. Hwang, et al., Open Access Dataset for EEG plus NIRS Single-Trial Classification, IEEE Trans. Neural Syst. Rehab. Eng., 25 (2017), 1735-1745. doi: 10.1109/TNSRE.2016.2628057
![]() |
1. | Lina Qiu, Yongshi Zhong, Zhipeng He, Jiahui Pan, Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning, 2022, 16, 1662-5161, 10.3389/fnhum.2022.973959 | |
2. | Yunyuan Gao, Biao Jia, Michael Houston, Yingchun Zhang, Hybrid EEG-fNIRS Brain Computer Interface Based on Common Spatial Pattern by Using EEG-Informed General Linear Model, 2023, 72, 0018-9456, 1, 10.1109/TIM.2023.3276509 | |
3. | Neela Rahimi, Chetan Kumar, John McLinden, Sarah Ismail Hosni, Seyyed Bahram Borgheai, Yalda Shahriari, Ming Shao, Topology-Aware Multimodal Fusion for Neural Dynamics Representation Learning and Classification, 2024, 24, 1530-437X, 21062, 10.1109/JSEN.2024.3400006 |
TIME(s) | Modality Type | |
EEG | fNIRS | |
0–3 | 78.9 | 67.2 |
1–4 | 80.3 | 69.1 |
2–5 | 78.1 | 72.0 |
3–6 | 79.2 | 75.0 |
4–7 | 79.4 | 78.0 |
5–8 | 80.2 | 80.9 |
6–9 | 79.4 | 83.4 |
7–10 | 75.7 | 85.9 |
Mean ± std | 78.90 ± 1.47 | 76.44 ± 6.77 |
Subject | Methods | |
STW | CTW | |
S1 | 88.2 | 92.5 |
S2 | 88.0 | 86.3 |
S3 | 92.5 | 91.8 |
S4 | 76.2 | 79.7 |
S5 | 78.0 | 80.2 |
S6 | 91.5 | 92.7 |
S7 | 85.2 | 92.5 |
S8 | 93.3 | 96.3 |
S9 | 85.3 | 86.8 |
S10 | 99.0 | 99.5 |
Mean ± std | 87.72±6.95 | 89.83±6.48 |
Subject | Methods | ||
Fisher | LASSO | FLFS | |
S1 | 93.2 | 92.6 | 93.3 |
S2 | 89.6 | 87.7 | 90.2 |
S3 | 92.3 | 95.5 | 95.7 |
S4 | 84.3 | 83.2 | 85.8 |
S5 | 81.0 | 80.3 | 83.3 |
S6 | 93.3 | 95.7 | 95.9 |
S7 | 92.5 | 92.3 | 93.2 |
S8 | 98.2 | 97.0 | 98.5 |
S9 | 85.8 | 88.7 | 89.3 |
S10 | 100.0 | 100.0 | 100.0 |
Mean ± std | 91.02 ± 5.97 | 91.30 ± 6.27 | 92.52 ± 5.38 |
TIME(s) | Modality Type | |
EEG | fNIRS | |
0–3 | 78.9 | 67.2 |
1–4 | 80.3 | 69.1 |
2–5 | 78.1 | 72.0 |
3–6 | 79.2 | 75.0 |
4–7 | 79.4 | 78.0 |
5–8 | 80.2 | 80.9 |
6–9 | 79.4 | 83.4 |
7–10 | 75.7 | 85.9 |
Mean ± std | 78.90 ± 1.47 | 76.44 ± 6.77 |
Subject | Methods | |
STW | CTW | |
S1 | 88.2 | 92.5 |
S2 | 88.0 | 86.3 |
S3 | 92.5 | 91.8 |
S4 | 76.2 | 79.7 |
S5 | 78.0 | 80.2 |
S6 | 91.5 | 92.7 |
S7 | 85.2 | 92.5 |
S8 | 93.3 | 96.3 |
S9 | 85.3 | 86.8 |
S10 | 99.0 | 99.5 |
Mean ± std | 87.72±6.95 | 89.83±6.48 |
Subject | Methods | ||
Fisher | LASSO | FLFS | |
S1 | 93.2 | 92.6 | 93.3 |
S2 | 89.6 | 87.7 | 90.2 |
S3 | 92.3 | 95.5 | 95.7 |
S4 | 84.3 | 83.2 | 85.8 |
S5 | 81.0 | 80.3 | 83.3 |
S6 | 93.3 | 95.7 | 95.9 |
S7 | 92.5 | 92.3 | 93.2 |
S8 | 98.2 | 97.0 | 98.5 |
S9 | 85.8 | 88.7 | 89.3 |
S10 | 100.0 | 100.0 | 100.0 |
Mean ± std | 91.02 ± 5.97 | 91.30 ± 6.27 | 92.52 ± 5.38 |