Features gradient-based signals selection algorithm of linear complexity for convolutional neural networks

1.   Introduction

Recently, many convolutional neural networks (CNNs) have been developed for classification in the time domain of signals, e.g., recognizing movement-intention using electroencephalography (EEG) signals ^[1], human activity recognition using inertial signals ^[2,3], and swimming stroke detection using acceleration signals ^[4]. Although these studies used all signals, signals that are not important for correct classification may be included in the CNN input layer. These signals worsen the accuracy of the classification and increase calculation costs, required memory, and data collection costs. Therefore, finding and removing non-important signals from the CNN input layer and creating a classification model for the minimum signals possible are crucial. However, many CNNs use all signals ^[4,5,6] or manually select signals ^[7,8].

We can visually find unimportant regions using gradient-weighted class activation mapping (grad-CAM) embedded in CNNs. Grad-CAM ^[9] uses the gradient of the feature maps output from the convolutional layer immediately before the fully connected layer to visualize which regions on the image influenced class estimation in the form of a heat map. For example, Figure 1 (A) is the generation process of grad-CAM when the image of 2 is input to CNN. First, the gradient of the output vector with respect to the feature maps is computed. Then, the feature maps are weighted by the sum of the gradients and they are stacked. A heat map is generated by passing the weighted feature map through the ReLu function. Considering this, we can understand that the CNN used the upper side of the image to estimate "2". Grad-CAM was proposed by improving CAM ^[10]. CAM is the white-box method for only CNNs, which do not have fully connected layers. In contrast, Grad-CAM is applicable to various CNN model-families ^[9]. The method is primarily used to input image data, e.g., chest CT images ^[11], 3D positron emission tomography images ^[12] and chest X-ray images ^[13,14,15].

Cases of applying grad-CAM to CNNs that input time domain data of signals, as shown in Figure 1 (B), are continuously increasing. For example, the classification of sleep stages ^[16], prognostication of comatose patients after cardiac arrest ^[17], classification of schizophrenia and healthy participants ^[18], and classification of motor imagery ^[19] are performed using EEG signal(s). Electrocardiogram (ECG) signals are used to detect nocturnal hypoglycemia ^[20] and predict 1-year mortality ^[21]. Acceleration signals have been used to detect hemiplegic gait ^[22] and human activity recognition ^[23].

From these studies, we visually find unimportant signals for correct classification by applying grad-CAM to CNN, inputting the time domain of the signals. In the example shown in Figure 1 (B), signals $\mathrm{s}_1$ and $\mathrm{s}_n$ are strongly activated, but signal $\mathrm{s}_2$ is not. Therefore, $\mathrm{s}_2$ can be interpreted as not important for class estimation. However, observing all of them and finding unimportant signals is extremely difficult because one grad-CAM for one input data is generated. Therefore, we propose features gradient-based signals selection algorithm (FG-SSA), which can be used to find and remove unimportant signals from a CNN input layer by utilizing features gradient obtained by the calculation process of grad-CAM. The algorithm provides a signal subset consisting of only important signals. The computational complexity of the proposed algorithm is linear order $\mathcal{O}(n_ \mathrm{s})$ (i.e., FG-SSA has a low calculation cost) when we define $n_ \mathrm{s}$ as the number of all signals:

The academic contributions of this study are twofold, which are specified as follows.

● We propose a method for quantifying the effect of signals on classes estimation.

● We propose an algorithm to remove signals of low importance.

This algorithm can remove unnecessary signals and enable CNNs and datasets to be smaller in size.

2.   Related works

The proposed algorithm, FG-SSA, is applied to CNNs that estimate class labels. It selects signals to be used in the input layer. This is similar to the feature selection problem of which features to use among all the features. The large number of patterns in feature combinations makes it difficult to find a globally optimal solution that maximizes verification performance for a subset of features. For example, a combination of 10 out of 100 features exceeds $10^{14}$ patterns. Therefore, the feature selection problem is known to be NP-hard ^[24,25]. For NP-hard problems, finding a good feasible solution is better than finding a global optimal solution ^[26]. Therefore, many methods for finding approximate solutions to the feature selection problem have been proposed, e.g., within-class variance between-class variance ratio ^[27,28], classification and regression tree ^[29], Lasso ^[30], ReliefF ^[31,32], out-of-bag error ^[33], and others. Recently, some studies have used Bayesian optimization for feature selection ^[34,35,36], which expresses whether to use a feature as a binary variable and evaluates the combination of features. These feature selection methods reduce the computation time of machine learning and improve the prediction accuracy ^[37,38].

The solution to the signal selection problem in CNNs also has a large number of patterns, which is obtained through several signal combinations. However, the feature selection algorithm described above cannot be used for signal selection problems and is not suitable for CNNs. Compared to other machine learning methods (support vector machine, decision tree, k-nearest neighbor, and others), CNNs require more computation time to construct a model. In addition, CNNs have many hyperparameters (number of layers, size of kernel filters, etc.), which also require computational resources for tuning. Therefore, instead of searching for an appropriate signals set, all of the prepared signals are often used such as ^[39,40]. Hence, a method that can find an approximate solution at a small computational cost is needed for the feature selection and signal selection problems.

Several studies have attempted to address the signal selection problem. Some studies applied Bayesian optimization to signal selection ^[41,42]. Because Bayesian optimization is a method for finding optimal solutions in black box functions, it can be applied to both feature selection and signal selection. However, it is not suitable for CNNs. Another approach is signal selection by group Lasso ^[43]. These signal selection algorithms are important research, but they are difficult to visually interpret because they do not utilize Grad-CAM. To the best of the authors' knowledge, only these previous studies involved signal selection. We feel that developing signal selection algorithms for CNNs is an important issue but has not been sufficiently studied. Therefore, we propose a signal selection algorithm, FG-SSA, that provides a visual interpretation of which signals contribute to which class estimation and is computationally inexpensive.

3.   Proposed method

3.1.   Initial signals set

We consider a situation in which the task is to estimate a class $c$ belonging to class set $\mathit{\boldsymbol{C}}$ defined by

where $n_ \mathrm{cm}$ is the number of classes. The measurement data of the initial signal set is defined by

where $n_ \mathrm{s}$ denotes the number of signals and $s_i$ denotes the signal identification names. Some of the $n_ \mathrm{s}$ signals are important for classification, whereas others are not. Therefore, finding a signal subset $\mathit{\boldsymbol{S}}_ \mathrm{use}$ that has removed the unimportant signals from all signal sets $\mathit{\boldsymbol{S}}$ is crucial. In this study, we provide an algorithm for finding such a subset of signals $\mathit{\boldsymbol{S}}_ \mathrm{use} \subseteq \mathit{\boldsymbol{S}}$. The applicable targets of the proposed method are all tasks of solving classification problems using CNNs by inputting the time domain of multisignals.

3.2.   Class estimation

An example of CNN with multiple signals as input is shown in Figure 2. This figure shows the CNN solving the problem of classifying 10 human motion labels from 15 acceleration signals mentioned in Subsection 4.2; however, the proposed algorithm can be applied to various problems. The data for the input layer are in the form of a matrix $w \times n_ \mathrm{s}$. $w$ is the window length of the sliding window method ^[44] and $n_ \mathrm{s} = |\mathit{\boldsymbol{S}}|$ is the number of signals.

Subsequently, the input data are convoluted using kernel filters of the time-directional convolution size $s_ \mathrm{c}$. The number of generated feature maps is $n_ \mathrm{f}$ because the number of filters is $n_ \mathrm{f}$. The reason for adopting only time-directional convolution is to avoid mixing a signal and others in the convolution process. The input data are convoluted by these $n_ \mathrm{c}$ convolution layers, and the CNN generates the feature maps shown in "extracted feature maps" in Figure 2. Subsequently, the CNN generates the output vector using fully connected layers and the SoftMax function. We define the output vector $\mathit{\boldsymbol{y}}$ as

The estimation class $c^{ \prime}$ is

Herein, we only define the output vector $\mathit{\boldsymbol{y}}$ because it appears in the grad-CAM definition. We explain grad-CAM for time domain signals in the next subsection.

3.3.   Time-directional grad-CAM

We defined the vertical size of the feature map before the fully connected layer as $s_ \mathrm{f}$, as shown in Figure 2. Let us denote the $k$th feature map of signal $s$ as

In the case of CNNs for image-data-based classification, the form of feature maps is a matrix. However, in the case of CNNs consisting of time-directional convolution layers, the form of the feature maps is a vector. We defined the effect of the feature map $\mathit{\boldsymbol{f}}^{s, k}$ on the estimation class $c^{ \prime}$ as

Then, we applied the grad-CAM of signal $s$ to estimation class $c^{ \prime}$ in the vector form of

The activated region of each signal can be understood by calculating $\mathit{\boldsymbol{Z}}^s_{c^{ \prime}}$ for all signals $\forall s \in \mathit{\boldsymbol{S}}$.

In this study, we refer to $\mathit{\boldsymbol{Z}}^s_{c^{ \prime}}$ as "time-directional grad-CAM" because it is calculated by the summing partial differentiations of the time direction. This was defined by a minor change in the basic grad-CAM for image data ^[9]. In the case of the basic grad-CAM ^[9], the CNN generates one grad-CAM for one input data. In contrast, in the case of time-directional grad-CAM, CNN generates grad-CAMs as many as the number of signals $n_ \mathrm{s}$ for one input data.

Figure 3 shows the examples of $\mathit{\boldsymbol{Z}}^s_{c^{ \prime}}$ calculated using the CNN presented in Figure 2 and the dataset described in Subsection 4.2. From top to bottom, these results correspond to $c^ \prime = \text{Stand N}, \text{Stand L}, \text{Stand R}$, and $\text{Walk N}$. Notable, an estimation class $c^ \prime$ is an element of $\mathit{\boldsymbol{C}}$. We can determine the signals that are important by viewing the time-directional grad-CAM shown in Figure 3. For example, signals from the left arm and left shoe are not used for estimating the Stand N class. Moreover, left arm signals are not used for the estimation of the Stand R class. Therefore, the time-directional grad-CAM is effective for finding unimportant signals.

3.4.   Signals importance index

Although we can find unimportant signals by viewing time-directional grad-CAM, the result varies for each input data. Therefore, viewing all the grad-CAMs is difficult when the data size is large. Herein, we quantify the importance of signal $s$ for classification based on $\alpha^{s, k}_{c^{ \prime}}$ defined in Eq (3.6).

Here, we denote the input dataset of size $n_\text{dat}$ as

The size of $i$th input data $\mathit{\boldsymbol{X}}_i$ is $w \times n_ \mathrm{s}$, which is the window length $w$ and number of signals $n_ \mathrm{s}$. When we define the input dataset of estimation class $c^ \prime \in \mathit{\boldsymbol{C}}$ as $\mathit{\boldsymbol{X}}_{c^ \prime}$, we can represent the input dataset $\mathit{\boldsymbol{X}}$ as

Using the set $\mathit{\boldsymbol{X}}_{c^ \prime}$, we define the importance of the signal $s \in \mathit{\boldsymbol{S}}$ to the estimation class $c^ \prime \in \mathit{\boldsymbol{C}}$ as

where

In addition, $\alpha^{s, k}_{c^{ \prime}}(\mathit{\boldsymbol{X}}_i)$ is $\alpha^{s, k}_{c^{ \prime}}$ of the input data $\mathit{\boldsymbol{X}}_i$ to a CNN, and $c^{ \prime}$ is the estimated class. Therefore, $L^{s}(\mathit{\boldsymbol{X}}_{c^ \prime})$ represents the importance of signal $s$ to class $c^{ \prime}$ based on the grad-CAM. Notably, we ignore terms with negative partial derivatives to extract the positive effect on classification, as shown in Eq (3.11). Moreover, using $L^{s}(\mathit{\boldsymbol{X}}_{c^ \prime})$, we can obtain the matrix

As shown in Eq (3.9), since $\mathit{\boldsymbol{X}} = \mathit{\boldsymbol{X}}_{c_1} \cup \mathit{\boldsymbol{X}}_{c_2} \cup \cdots \cup \mathit{\boldsymbol{X}}_{c_{n_\mathrm{cm}}}$ is satisfied, we define

We refer to $\mathit{\boldsymbol{I}}_ \mathrm{mat}(\mathit{\boldsymbol{X}})$ as the "signals importance matrix (SIM)" because the matrix comprises the importance of all signals and classes using the input dataset $\mathit{\boldsymbol{X}}$. The effect of each signal to all classes can be understood by calculating and viewing SIM $\mathit{\boldsymbol{I}}_ \mathrm{mat}(\mathit{\boldsymbol{X}})$.

Although $\mathit{\boldsymbol{I}}_ \mathrm{mat}(\mathit{\boldsymbol{X}})$ includes important information, summarizing the SIM is necessary to find signals that are unimportant to all classes. Therefore, by calculating the summation of the row values of SIM, we denote the importance of signal $s$ as

where $I^{s, c^ \prime}_ \mathrm{mat}(\mathit{\boldsymbol{X}})$ is the value of row $s$ and column $c^ \prime$ of $\mathit{\boldsymbol{I}}_ \mathrm{mat}(\mathit{\boldsymbol{X}})$. By calculating $I^s(\mathit{\boldsymbol{X}})$ for all signals $s$, we obtain the vector as

We refer to $\mathit{\boldsymbol{I}}_ \mathrm{vec}(\mathit{\boldsymbol{X}})$ as the "signals importance vector (SIV)" because it is the vector that consists of the signal importance.

The calculated examples of SIM and SIV are shown in Figure 4. This is the effect of the each individual signal on the class estimation by the CNN. These values were calculated using the CNN for Condition A described in Section 5. The result of SIM (columns 1 to 10 in Figure 4) includes various important data. First, we can confirm that the Right arm X signal is used for right hand movement (Stand R, Walk R, and Sit R) estimation. In the case of the left hand movement (Stand L, Walk L) estimation, the signals from the Back sensor responds strongly. Since the Back sensor is attached to the body trunk, it is assumed that the body trunk movement is used to estimate the movement of the left hand. The Left arm X responds to another left hand movement (Sit L) estimation. In addition, the Left shoe Y and Right shoe Y respond strongly to the motion estimation of Walk N. Based on these results, the SIM is considered reasonable. However, since some results are not comfortable, it is important to remove unnecessary signals by FG-SSA.

Moreover, we can identify the signals that are important for all classes by viewing SIV (the column 11 in Figure 4). From SIV, we denote the minimum importance signal $s_ \mathrm{min}$ and maximum signal $s_ \mathrm{max}$ as

We expect to maintain the estimation accuracy even when removing $s_ \mathrm{min}$ because it is a minimum importance signal. In contrast, we expect the accuracy to decrease when we remove $s_ \mathrm{max}$ and re-learn the CNN because it is the most important signal.

3.5.   Signals selection algorithm

In this study, we proposed an algorithm to find a desirable signal subset $\mathit{\boldsymbol{S}}_ \mathrm{use} \subseteq \mathit{\boldsymbol{S}}$ by removing unimportant signals. The proposed method is presented in Algorithm 1. The main inputs to the algorithm are the training dataset $\mathit{\boldsymbol{X}}_ \mathrm{train}$, validation dataset $\mathit{\boldsymbol{X}}_ \mathrm{valid}$, and input initial signals set $\mathit{\boldsymbol{S}}$. To avoid data leakage, the test dataset was not used in the algorithm. The elements of set $\mathit{\boldsymbol{S}}$ are the signal identification names defined in Eq (3.2). The first procedure is to create the initial signal set $\mathit{\boldsymbol{S}}_0$, which consists of all signals (line 1). The next step is to develop a CNN using the training dataset $\mathit{\boldsymbol{X}}_ \mathrm{train}^{\mathit{\boldsymbol{S}}_0}$, consisting of $\mathit{\boldsymbol{S}}_0$ (line 3). Subsequently, we measure the validation accuracy $A(\mathit{\boldsymbol{X}}_ \mathrm{valid}^{\mathit{\boldsymbol{S}}_0})$ using the validation dataset $\mathit{\boldsymbol{X}}_ \mathrm{valid}^{\mathit{\boldsymbol{S}}_0}$ (line 4). Next, we determine the most unimportant signal $s_ \mathrm{min} \in \mathit{\boldsymbol{S}}_0$ based on Eq (3.16) (line 5). Finally, we obtain the next signal set $\mathit{\boldsymbol{S}}_1$ by removing $s_ \mathrm{min}$ from $\mathit{\boldsymbol{S}}_0$ (line 6).

This procedure is repeated until the number of signals reaches one (i.e., $t = |\mathit{\boldsymbol{S}}| - 1$), and the validation accuracies are recorded. The algorithm returns the signal subset $\mathit{\boldsymbol{S}}_{ \mathrm{use}}$ leading to maximum validation accuracy (lines 8 and 9). Notably, the case wherein the signal subset leading to the maximum accuracy is the initial signal set can occur (i.e., $\mathit{\boldsymbol{S}}_{ \mathrm{use}} = \mathit{\boldsymbol{S}}$). In cases wherein the main purpose is to achieve maximum accuracy, adopting the initial signal set as an optimal subset is a better option, if $\mathit{\boldsymbol{S}}_{ \mathrm{use}} = \mathit{\boldsymbol{S}}$ occurs. However, some cases exist that require a decrease in the number of signals. Therefore, we prepared constraints wherein the number of adopted signal sizes is $\gamma$ or less (i.e., $|\mathit{\boldsymbol{S}}_{ \mathrm{use}}| \le \gamma$). This is a hyperparameter of the proposed algorithm.

When the size of the initial signal set $\mathit{\boldsymbol{S}}$ is $n_ \mathrm{s}$, the number of developing CNNs in Algorithm 1 is as follows:

That is, the computational complexity of Algorithm 1 is $\mathcal{O}(n_ \mathrm{s})$. This means that even if the signal size $n_ \mathrm{s}$ increases, the signal subset is returned in realistic time.

Generally, the total pattern for choosing $m$ from $n$ signals is ${}_{n} \mathrm{C}_{m}$. Because $m$, which is the number of signals leading to the maximum validation accuracy, is not known, the total number of combinations of input layers $T_ \mathrm{bs}(n_ \mathrm{s})$ is similar to

Therefore, finding the optimal signal subset using a brute-force search is difficult. The computation time tends to be large because the CNN includes other hyperparameters. From this viewpoint, a fast algorithm such as the proposed method is important.

4.   Experiment 1: Relationship between validation accuracy and the number of deleted signals

4.1.   Objective and outline

Herein, we examine the reliability of $s_ \mathrm{min, \ max}$ defined in Eq (3.16). We expect to maintain estimation accuracy even if the signal $s_ \mathrm{min}$ is removed because it is the most unimportant signal. In contrast, because signal $s_ \mathrm{max}$ is the most important signal, the estimation accuracy may decrease by removing it. We gradually removed signal $s_ \mathrm{min, \ max}$ using the processes of lines 2 to 7 of Algorithm 1 to verify the aforementioned hypothesis. In addition, we repeatedly developed CNNs and recorded their validation accuracies. We performed verification using a total of 100 seeds because the training process of CNN depends on randomness. The adopted layer structure of the CNN is explained in the Appendix section. The number of epochs was 300.

4.2.   Dataset

Herein, we define the initial signals set $\mathit{\boldsymbol{S}}$ and the classes set $\mathit{\boldsymbol{C}}$ described in Subsection 3.1 concretely with real data. We used the dataset "OPPORTUNITY Activity Recognition". This is the dataset for activity recognition, and it is used in "Activity Recognition Challenge" held by IEEE in 2011 ^[45,46]. The dataset is regarded as reliable because it is used for the performance evaluation of machine learning in some studies ^[47,48]. The dataset contains data on multiple inertial measurement units, 3D acceleration sensors, ambient sensors, and 3D localization information. Four participants performed a natural execution of daily activities. The activity annotations include locomotions (e.g., sitting, standing, and walking) and left- and right-hand actions (e.g., reach, grasp, release, and etc.). Details of the dataset are described in ^[45,46].

In this study, we used five triaxial acceleration sensors from all the measurement devices (sampling frequency: 32 [Hz]). Attachment points of sensors on the human body are "back, " "right arm, " "left arm, " "right shoe, " and "left shoe." The total number of signals was 15 because we adopted five triaxial sensors ($5 \times 3 = 15$). We adopted data splitting using the sliding window method with window length $w = 60$ and sliding length $60$. One signal length was approximately 2 s since the sampling frequency was 32 [Hz]. Searching for the optimal window length size is important because it is a hyperparameter that affects the estimation accuracy ^[44,49,50]. However, we did not tune the window length size because the aim of this study was to provide a signal-selection algorithm.

Subsequently, we labeled the motion class label for each dataset based on human activity. The class labels are combinations of locomotions ("Stand, " "Walk, " "Sit, " and "Lie") and three hands activity ("R": moving right hand, "L": moving left hand, and "N": not moving hands). For example, the class label "Sit R" refers to the sitting and right-hand motions.

Table 1 lists the results of applying the described procedures to all the data. This indicates that the data size of the left-hand motion is small. Nearly all participants were right-handed (notably, we could not find a description of the participants' dominant arms in the explanation of the OPPORTUNITY dataset). Moreover, data belonging to Lie R and L are absent. Therefore, these classes were removed from the estimation task (i.e., the total number of classes was 10).

Subsequently, we randomly split all the data into a training dataset (80%) and a test dataset (20%). Moreover, we assigned 20% of the training dataset to the validation dataset. The training, validation, and test datasets were independent because we adopted the sliding window method for the same window and slide length (60 steps). We defined the signal set $\mathit{\boldsymbol{S}}$, which has 15 elements, and the class set $\mathit{\boldsymbol{C}}$, which has 10 elements, as follows:

In other words, CNNs solve 10 classification problems from 15 multisignals. Moreover, an algorithm for removing unimportant signals was provided while maintaining the estimation accuracy. Although the sets $\mathit{\boldsymbol{S}}$ and $\mathit{\boldsymbol{C}}$ represent acceleration signals and human activities, respectively, the proposed algorithm can be used for other diverse signals, such as EEG and ECG.

4.3.   Result and discussion

First, we indicate the validation accuracies when the most unimportant signal, $s_ \mathrm{min}$, is gradually removed, as shown in Figure 5 (A). From left to right in Figure 5 (A), the number of deleted signals increases (i.e., the number of signals in the CNN input layer decreases). The leftmost and rightmost results are obtained using all signals and only one signal as the CNN input layer, respectively. The results show that even if six signals were deleted, the validation accuracy did not decrease. Moreover, the accuracy significantly decreases by removing seven or more signals. In this case, although CNNs estimate class labels from 15 signals in the set $\mathit{\boldsymbol{S}}$, six signals are unnecessary.

We calculated the average removed timings of the 15 signals to determine the unnecessary signals. Figure 5 (B) shows the results. This means that the lower the value, the earlier the signal is removed in the procedure of Algorithm 1. The results indicate that the signals of the right and left shoes are removed earlier than those of the other sensors. In contrast, some signals from the back, right arm, and left arm did not disappear early. Therefore, we can regard shoe signals as unimportant for classification. In addition, shoe sensors are important for walk motion classification. However, even if shoe sensors disappear, we consider that the back sensor attached to centroids of the human body contributes to the classification of walk motions because these motions are periodic.

Then, we indicate the validation accuracies when the most important signal, $s_ \mathrm{max}$, is gradually removed in Figure 6 (A). We can verify that the validation accuracy statistically decreases by removing one of the most important signals. Therefore, removing the most important signal, $s_ \mathrm{max}$, leads to a worse accuracy. The average timing of the signal removal is shown in Figure 6 (B). From this figure, we confirm that the signals of back X, right arm X, and left arm Y are removed early. Moreover, we confirm that shoe sensor signals are not removed early. These tendencies are in contrast compared with the case of removing the most unimportant signal $s_ \mathrm{min}$.

Clearly, (1) even if we remove the most unimportant signal $s_ \mathrm{min}$, the estimation performance remains, and (2) the performance decreases when we remove the most important signal $s_ \mathrm{max}$. Therefore, the signal importance $I^s(\mathit{\boldsymbol{X}})$ defined in Eq (3.14) is reliable.

5.   Experiment 2: Effectiveness of FG-SSA on generalization scores

5.1.   Objective and outline

Herein, we confirmed the effect of FG-SSA indicated in Algorithm 1 on the generalization performance of the classification. Therefore, we developed the following three conditions for CNNs:

● Condition A: CNN using all signals (i.e., FG-SSA is not used).

● Condition B: CNN for applied FG-SSA of $\gamma = n_ \mathrm{s}$.

● Condition C: CNN for applied FG-SSA of $\gamma = 9$.

Condition A means that the CNN does not remove signals but uses all signal sets $\mathit{\boldsymbol{S}}$. Condition B refers to the CNN using the signal subset $\mathit{\boldsymbol{S}}_ \mathrm{use}$ obtained by the FG-SSA, given $n_ \mathrm{s}$ as the constraint parameter $\gamma$. Under this condition, we allow the algorithm to return $\mathit{\boldsymbol{S}}$ as $\mathit{\boldsymbol{S}}_ \mathrm{use}$ when the maximum validation accuracy is achieved using all signal sets $\mathit{\boldsymbol{S}}$. In other words, a case wherein no signals are removed can occur. Condition C implies that the number of adopted signals is nine or less for a CNN input layer (i.e., $|\mathit{\boldsymbol{S}}_ \mathrm{use}| \le 9$). In other words, six or more signals were deleted because the initial number of signals $n_ \mathrm{s}$ was 15. The value of hyper-parameter $\gamma = 9$ was determined by referring to the result shown in Figure 5 (A).

We developed CNNs using the signal set, which was determined by FG-SSA. Moreover, the optimal epochs leading to a maximum validation accuracy were adopted. The search range of epochs was from 1 to 300. Subsequently, the generalization performance is measured using the test dataset. The test dataset was used only at this time (i.e., it was not used for the parameter search). We developed CNNs for Conditions A, B, and C with a total of 100 seeds to remove the randomness effect.

5.2.   Result and discussion

By developing CNNs under Conditions A, B, and C on a total of 100 seeds, the average number of signals used was 15.00, 11.94, and 8.35, respectively. In other words, an average of 3.06 and 6.65 signals were removed by FG-SSA in the cases of Conditions B and C, respectively. The generalization performance (F score, precision, and recall) measured by the test dataset for each condition is shown in Figure 7. These are histograms of 100 CNNs developed using 100 random seeds.

The results indicated that the generalization scores were nearly the same for Conditions A, B, and C. Moreover, $p$ values of the two-sided $t$-test were not statistically significant. The confusion matrices obtained by CNNs for each condition are shown in Figure 8. The values were averaged by 100 seed results and standardized, where the summation in each row was 1. Consequently, the confusion matrices were nearly the same. Although the proposed algorithm removed some signals, generalization errors did not increase. Therefore, we consider that FG-SSA can find and remove signals that are not important for CNN-based classification.

From another viewpoint, the number of correct classifications of left hand motions (Stand L, Walk L, and Sit L) was small in all conditions. We adopted weighted cross-entropy-based learning for CNNs because the original data size of the left hand motions is small, as shown in Table 1. However, we consider that an appropriate classification cannot be performed because the data diversity of these motions is low. Although we believe that nearly all participants are right-handed, the explanation of participant's dominant arm in the OPPORTUNITY dataset is insufficient to the best of our knowledge.

Note that this study also validates FG-SSA on a dataset other than OPPORTUNITY. This dataset is for an experiment that classifies swimming styles with signals measured from a single inertial sensor. The details of this experiment and the results obtained are presented in Appendix B. The experimental results also indicate that FG-SSA is effective at signal selection.

6.   Comparison of FG-SSA and other methods

6.1.   Objective and outline

From the described results, it was confirmed that signals can be removed without degrading the generalization performance using FG-SSA. In this section, we compare the generalization performance obtained by signal subsets selected by Condition C mentioned in Section 5 and signal subsets obtained by other signal selection methods. Then, we evaluate the effectiveness of the proposed method. Since Condition C has a problem of selecting a maximum of nine signals from 15 signals, this condition was also adopted in existing methods.

Bayesian optimization (BO) and random search (RS) were adopted as existing methods. BO is a search algorithm that solves combinatorial optimization problems and is sometimes applied to hyperparameter tuning in machine learning and deep learning ^[51,52]. In this approach, the next hyperparameters to be verified are determined based on validation accuracies obtained with the previously set hyperparameters. This can be applied to feature selection problems in classification problems ^[34,35,36]. Feature selection is a method of selecting features to be used for classification from all features prepared in advance. This is similar to the method of choosing signal subset from all signals. Therefore, BO can also be applied to signal selection and has been applied in several studies ^[41,42]. Therefore, in this study, we adopted BO as an existing method for comparison with FG-SSA.

As shown in Eq (3.18), when selecting some signals from $n_ \mathrm{s}$ signals, the number of selectable combinations becomes enormous. In contrast, as shown in Algorithm 1, FG-SSA can obtain the signal subset to be used only by constructing $n_ \mathrm{s}$ CNNs. On the other hand, the greater the number of iterations for BO, the higher the possibility of discovering a desirable signal subset. However, in comparing FG-SSA and BO, it is fairer to set the number of CNN constructions to be the same. Therefore, we set the number of iterations of BO to $n_ \mathrm{s}$. Note that $n_ \mathrm{s}$ is 15 in this experiment, as mentioned in Section 5. We adopted tree-structured Parzen estimator (TPE) ^[53] as a specific method of BO and Optuna ^[54] as a framework for implementation. TPE has the property of trying to obtain the desired solution with a small number of iterations ^[55].

Moreover, RS was adopted as a baseline. This is a method that repeats the process of randomly selecting signals and constructing a CNN for a specified number of times and uses the signal subset with the highest validation accuracy. This time, we decided to repeat the process of selecting up to nine signals from 15 signals $n_ \mathrm{s}$ times. In other words, the numbers of constructing CNNs for RS, BO, and FG-SSA are the same.

The details of the process of existing methods are described in Algorithm 2. The inputs for this algorithm are the training dataset $\mathit{\boldsymbol{X}}_ \mathrm{train}$, validation dataset $\mathit{\boldsymbol{X}}_ \mathrm{valid}$, initial signals set $\mathit{\boldsymbol{S}}$ defined in Eq (4.1), maximum number of selecting signals $S_ \mathrm{max}$, number of iterations $T$, and searching method $M \in \{ \mathrm{BO}, \mathrm{RS}\}$. BO means the TPE-based Bayesian optimization and RS means random search. In Algorithm 2, the signal subset $\mathit{\boldsymbol{S}}_t$ consisting of $S_ \mathrm{max}$ signals is extracted by "SignalsSelection" function (line 2 in Algorithm 2, BO or RS method). We developed a CNN using the training dataset $\mathit{\boldsymbol{X}}_ \mathrm{train}^{\mathit{\boldsymbol{S}}_t}$ consisting of the signals subset $\mathit{\boldsymbol{S}}_t$ and measured the validation accuracy $A(\mathit{\boldsymbol{X}}_ \mathrm{valid}^{\mathit{\boldsymbol{S}}_t})$ using the validation dataset $\mathit{\boldsymbol{X}}_ \mathrm{valid}^{\mathit{\boldsymbol{S}}_t}$. We repeated it from $t = 1$ to $t = T$ (i.e., $T$ times) and selected the signal subset leading to the maximum validation accuracy. This subset is $\mathit{\boldsymbol{S}}_ \mathrm{use} \subseteq \mathit{\boldsymbol{S}}$ and it is returned by Algorithm 2 to users. Then, we developed a CNN using the signal subset $\mathit{\boldsymbol{S}}_ \mathrm{use}$ and measured the generalization scores (F score, precision, and recall) using the test dataset. The layer structure and learning condition of the CNN are the same as the previous experiment described in Section 5.

In general, for both BO and RS, better signal subsets can be found as the number of iterations $T$ increases. However, CNN requires a long time to build; thus, it is not desirable to increase $T$. As shown in Eq (3.17), the strength of FG-SSA is that it is a fast algorithm. Therefore, we decided to make the number of iterations $T$ of BO and RS and the maximum number of selected signals $S_ \mathrm{max}$ the same as those of FG-SSA (Section 5, Condition C). In other words, we set $T = 15, S_ \mathrm{max} = 9$. We ran Algorithm 2 100 times while changing the random seed to remove the effect of randomness (FG-SSA in Condition C is also the same).

6.2.   Result and discussion

The generalization scores of FG-SSA, BO, and RS measured by the test dataset are shown in Table 2. The results consist of F score, precision, and recall of 10 classes belonging to the class set $\mathit{\boldsymbol{C}}$ defined by Eq (4.2). The bold and underlined numbers indicate the highest and lowest scores, respectively. The values in the cells indicate the average scores of 100 runs, and $p$ values are the result of the two-sided $t$ test between FG-SSA and BO.

Based on Table 2, the baseline RS has the lowest score. A possible reason for this is that random signal selection is not strategic in terms of improving the generalization performance. In addition, FG-SSA has obtained better scores in more classes than BO. Although some results have better scores with BO, there is no significant difference in comparison with FG-SSA. Additionally, looking at the macro averages, we confirmed that FG-SSA scores higher than BO and RS in all three indicators. Considering this, it can be interpreted that FG-SSA can discover signal subsets leading to higher generalization performance than BO and RS. Therefore, FG-SSA is an excellent method from the viewpoint of performing good signal selection with a small number of iterations.

Since the number of CNN constructions in FG-SSA is 15, the number of Bayesian optimization iterations was set to 15 in the previous analyses. As shown in Table 2, FG-SSA demonstrated higher estimation performance than BO. However, by increasing the number of iterations of BO, it may be possible to find a signal set that yields scores similar to those of FG-SSA. To test this hypothesis, the generalization performance was measured when the number of BO iterations was 15, 30, and 45 (BO15, BO30, and BO45). Table 3 shows the obtained results. It can be seen that the generalization performance increases slightly as the number of BO iterations increases. When comparing FG-SSA and BO15, FG-SSA performed significantly higher; however, when comparing FG-SSA and BO45, no significant differences were observed for all indices. Therefore, the two signal sets obtained with BO45 and FG-SSA could be similar. Conversely, the computation time for BO45 was approximately three times longer than that for FG-SSA. Therefore, FG-SSA can find a good signals set at a lower computational cost than BO, which is an existing method. Note that the computational environment used in this experiment was OS: Ubuntu 20.04 LTS, CPU: Xeon W-2225 (4.1 GHz), RAM: 32 GB DDR4-2933, GPU: NVIDIA RTX A6000 (48 GB).

7.   Conclusion, limitations, and future works

In this study, we described the following topics to find and remove unimportant signals for CNN-based classification:

● (1) The signal importance indices SIM $\mathit{\boldsymbol{I}}_ \mathrm{mat}(\mathit{\boldsymbol{X}})$ and SIV $\mathit{\boldsymbol{I}}_ \mathrm{vec}(\mathit{\boldsymbol{X}})$ are explained in Subsection 3.4.

● (2) The algorithm of the linear complexity $\mathcal{O}(n_ \mathrm{s})$ for obtaining the signals subset $\mathit{\boldsymbol{S}}_ \mathrm{use}$ from the initial signals set $\mathit{\boldsymbol{S}}$ by finding and removing unimportant signals (see Algorithm 1).

Although the proposed algorithm performed well in the case of the OPPORTUNITY dataset, it had some limitations. This is explained in the following sections. Future work will confirm these findings.

● (A) The results described in this paper were obtained from limited cases. Although we assume that the algorithm can be applied to data other than acceleration signals (e.g., EEG and ECG), it is not validated.

● (B) In this experiment, the initial signals set size was 15. The number of signals may be much higher depending on some situations. It is important to determine the relationship between the number of unimportant signals and the effectiveness of FG-SSA.

● (C) We assume that the algorithm may extract the signals' subset weighted specific target class by providing class weights to the signal importance $I^{s}(\mathit{\boldsymbol{X}})$ defined in Eq (3.14). In other words, we denote the importance of signal $s$ weighted classes as

where $w^{c^ \prime}$ is the weight of class $c^ \prime$ and $\mathit{\boldsymbol{w}}$ is the class weight vector, which is defined as

We note that the algorithm based on $I^s(\mathit{\boldsymbol{X}}; \mathit{\boldsymbol{w}})$ can return the signals' subset weighted specific classes. However, this effect is not confirmed in this study. This will be studied in the future.

A.   Appendix: Layers structure of CNNs

The CNN's layer structure is illustrated in Figure 2. First, an input layer exists for inputting $w \times n_ \mathrm{s}$ size data ($w$: window length and $n_ \mathrm{s}$: the number of signals). Then, $n_ \mathrm{c} = 3$ convolution layers exist to generate $n_ \mathrm{f}$ feature maps using kernel filters of convolution size $s_ \mathrm{c}$. The convolutions are only in the direction of time. Subsequently, the generated feature maps are transformed into a vector form by the flattened layer. Vector dimensions are gradually reduced to 200, 100, and 50. Finally, an output vector $\mathit{\boldsymbol{y}}$ of 10 dimensions is generated, and classification is performed. The total number of CNN layers is nine (one input layer, three convolution layers, four dense layers, and one output layer). The activation function of each layer is ReLU.

We performed a hyperparameter search for convolution size $s_ \mathrm{c}$, the number of kernel filters $n_ \mathrm{f}$, and learning rate $r$. Particularly, CNNs were trained using the training and validation datasets described in Subsection 3.1 and 300 epochs. We adopted a weighted cross-entropy-based loss function based on the inverse values of class sample sizes because the sample sizes of each class are imbalanced, as shown in Table 1.

The parameter candidates are as follows:

The combination of hyperparameters leading to the maximum validation accuracy was $(s_ \mathrm{c}, n_ \mathrm{f}, r) = (10, 10, 10^{-4})$ (accuracy: 0.726). Therefore, the adopted CNN architecture is as follows:

● Layer 1: Conv. layer, the kernel filter: $(s_ \mathrm{c}, n_ \mathrm{f}) = (10, 10)$

● Layer 2: Conv. layer, the kernel filter: $(s_ \mathrm{c}, n_ \mathrm{f}) = (10, 10)$

● Layer 3: Conv. layer, the kernel filter: $(s_ \mathrm{c}, n_ \mathrm{f}) = (10, 10)$

● Layer 4: Flatten layer based on the final conv. layer's output size

● Layer 5: Dense layer of 200 dimensions

● Layer 6: Dense layer of 100 dimensions

● Layer 7: Dense layer of 50 dimensions

● Layer 8: Classification layer

● Note: the activation function for all layers except the classification layer is ReLU.

We adopted this architecture for all CNNs used in this study. Although CNNs have many other hyperparameters, we avoided excessive tuning because the aim of this study was to provide a solution to the signal selection problem.

B.   Appendix: Effectiveness of FG-SSA on swimming style classification

B.1.   Outline

The effectiveness of the proposed method should be validated by multiple datasets, not only by a single dataset. Therefore, in this study, FG-SSA was applied to the task of automatically classifying swimming style (backstroke, breaststroke, butterfly, and front crawl) based on signals measured by an inertial sensor and CNN. Swimming style classification with inertial sensors is a typical research topic in sports engineering (e.g., ^[56,57]). The subjects were 16 Japanese university students who selected two swimming styles from four swimming styles. The pool had a length of 25 m, and each swimmer made one round trip of swimming (total 50 m). A sensor was attached to the lower back of the swimmers, and it measured the tri-axial acceleration and angular velocity (six signals in total). The sampling frequency of the inertial sensor was 100 Hz, the acceleration range was $\pm 5$ G, and the angular velocity range was $\pm 1500$ dps. All signals were standardized to a minimum of 0 and a maximum of 1. To verify whether FG-SSA can eliminate unnecessary signals and select only important signals, nine meaningless signals generated by uniform random numbers in the 0 to 1 range were prepared. Considering the nine meaningless signals, the total number of signals was $3+3+9 = 15$.

B.2.   Preprocessing

Of the 32 swims (16 subjects $\times$ 2 swims), there was a measurement error (sensor fell off while swims) in 5 swims. Therefore, usable data were available for 27 swimmings. The sliding window method was applied to the data with the window length and sliding length set to 1 sec. Since window length and sliding length are of the same size, the cut waveforms did not overlap. This process resulted in sample sizes of 91, 145, 332, and 205 for backstroke, breaststroke, butterfly, and front crawl, respectively. Of the available data, 80 % was used as training data and 20 % as test data. In addition, 20 % of the training data was used as validation data. Since the data splitting is random, the performance may be higher (or lower) than the average tendency due to random effects. To obtain reliable results, the experiment was run with a total of 100 random seeds.

B.3.   Result and discussion

Figure 9 (a) shows the maximum validation accuracies in 300 epochs when FG-SSA is applied to the aforementioned task and unnecessary signals are removed one by one. In the initial half of FG-SSA, the estimation performance improves as more unnecessary signals are deleted, reaching a maximum when the number of deletions is 9. Then, in the later half of FG-SSA, the estimation performance degrades. This indicates that the estimation performance is worse when there are unnecessary signals in the input layer or when important signals are removed. Figure 9 (b) shows the deletion timing of each signal. Randomly generated meaningless signals are removed early in the algorithm, and the tri-axial acceleration and angular velocity signals are removed later. This result indicates that FG-SSA can efficiently find and remove only meaningless signals while keeping important signals.

The number of signals deleted by processing line 8 of Algorithm 1 was $9.66 \pm 1.88$ signals (average and standard deviation of the results of 100 seeds). In other words, the number of signals selected as the input layer of the CNN by FG-SSA was $15 - 9.66 = 5.34$. Next, the generalization performance was measured by the test dataset for CNNs using all signals (Condition A, 15 signals) and for CNNs using the signal set selected by FG-SSA (Condition B, average 5.34 signals). The result is shown in Figure 10. When all signals were used (Condition A), confusion in the estimation of "breaststroke" and "butterfly" was observed. On the other hand, when unnecessary signals were removed by FG-SSA (Condition B), misclassification was greatly reduced. Therefore, it can be concluded that signal reduction by FG-SSA is beneficial for improving the generalization performance.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported in part by JSPS Grant-in-Aid for Scientific Research (C) (Grant Nos. 21K04535 and 23K11310), and JSPS Grant-in-Aid for Young Scientists (Grant No. 19K20062). This work was also supported by the Tokyo City University Prioritized Studies.

Conflict of interest

The authors declare no conflicts of interest.